Explore Developer Center's New Chatbot! MongoDB AI Chatbot can be accessed at the top of your navigation to answer all your MongoDB questions.

Join us at AWS re:Invent 2024! Learn how to use MongoDB for AI use cases.
MongoDB Developer
Python
plus
Sign in to follow topics
MongoDB Developer Centerchevron-right
Developer Topicschevron-right
Languageschevron-right
Pythonchevron-right

Introduction to Multi-Document ACID Transactions in Python

Joe Drumgoole10 min read • Published Feb 05, 2022 • Updated Sep 11, 2024
MongoDBPython
Facebook Icontwitter iconlinkedin icon
Rate this quickstart
star-empty
star-empty
star-empty
star-empty
star-empty

Introduction

QuickStart Python Logo
Multi-document transactions arrived in MongoDB 4.0 in June 2018. MongoDB has always been transactional around updates to a single document. Now, with multi-document ACID transactions we can wrap a set of database operations inside a start and commit transaction call. This ensures that even with inserts and/or updates happening across multiple collections and/or databases, the external view of the data meets ACID constraints.
To demonstrate transactions in the wild we use a trivial example app that emulates a flight booking for an online airline application. In this simplified booking we need to undertake three operations:
  • Allocate a seat in the seat_collection
  • Pay for the seat in the payment_collection
  • Update the count of allocated seats and sales in the audit_collection
For this application we will use three separate collections for these documents as detailed above. The code in transactions_main.py updates these collections in serial unless the --usetxns argument is used. We then wrap the complete set of operations inside an ACID transaction. The code in transactions_main.py is built directly using the MongoDB Python driver (Pymongo 3.7.1).
The goal of this code is to demonstrate to the Python developers just how easy it is to covert existing code to transactions if required or to port older SQL based systems.

Setting up your environment

The following files can be found in the associated github repo, pymongo-transactions.
  • gitignore : Standard Github .gitignore for Python.
  • LICENSE : Apache's 2.0 (standard Github) license.
  • Makefile : Makefile with targets for default operations.
  • transaction_main.py : Run a set of writes with and without transactions. Run python transactions_main.py -h for help.
  • transactions_retry.py : The file containing the transactions retry functions.
  • watch_transactions.py : Use a MongoDB change stream to watch collections as they change when transactions_main.py is running.
  • kill_primary.py : Starts a MongoDB replica set (on port 7100) and kills the primary on a regular basis. This is used to emulate an election happening in the middle of a transaction.
  • featurecompatibility.py : check and/or set feature compatibility for the database (it needs to be set to "4.0" for transactions).
You can clone this repo and work alongside us during this blog post (please file any problems on the Issues tab in Github).
We assume for all that follows that you have Python 3.6 or greater correctly installed and on your path.
The Makefile outlines the operations that are required to setup the test environment.
All the programs in this example use a port range starting at 27100 to ensure that this example does not clash with an existing MongoDB installation.

Preparation

To setup the environment you can run through the following steps manually. People that have make can speed up installation by using the make install command.

Set a python virtualenv

Check out the doc for virtualenv.
1$ cd pymongo-transactions
2$ virtualenv -p python3 venv
3$ source venv/bin/activate

Install Python MongoDB Driver pymongo

Install the latest version of the PyMongo MongoDB Driver (3.7.1 at the time of writing).
1pip install --upgrade pymongo

Install mtools

mtools is a collection of helper scripts to parse, filter, and visualize MongoDB log files (mongod, mongos). mtools also includes mlaunch, a utility to quickly set up complex MongoDB test environments on a local machine. For this demo we are only going to use the mlaunch program.
1pip install mtools
The mlaunch program also requires the psutil package.
1pip install psutil
The mlaunch program gives us a simple command to start a MongoDB replica set as transactions are only supported on a replica set.
Start a replica set whose name is txntest. See the make init_server make target for details:
1mlaunch init --port 27100 --replicaset --name "txntest"

Using the Makefile for configuration

There is a Makefile with targets for all these operations. For those of you on platforms without access to Make, it should be easy enough to cut and paste the commands out of the targets and run them on the command line.
Running the Makefile:
1$ cd pymongo-transactions
2$ make
You will need to have MongoDB 4.0 on your path. There are other convenience targets for starting the demo programs:
  • make notxns : start the transactions client without using transactions.
  • make usetxns : start the transactions client with transactions enabled.
  • make watch_seats : watch the seats collection changing.
  • make watch_payments : watch the payment collection changing.

Running the transactions example

The transactions example consists of two python programs.
  • transaction_main.py,
  • watch_transactions.py.

Running transactions_main.py

1$ python transaction_main.py -h
2usage: transaction_main.py [-h] [--host HOST] [--usetxns] [--delay DELAY]
3 [--iterations ITERATIONS]
4 [--randdelay RANDDELAY RANDDELAY]
5
6optional arguments:
7 -h, --help show this help message and exit
8 --host HOST MongoDB URI [default: mongodb://localhost:27100,localh
9 ost:27101,localhost:27102/?replicaSet=txntest&retryWri
10 tes=true]
11 --usetxns Use transactions [default: False]
12 --delay DELAY Delay between two insertion events [default: 1.0]
13 --iterations ITERATIONS
14 Run N iterations. O means run forever
15 --randdelay RANDDELAY RANDDELAY
16 Create a delay set randomly between the two bounds
17 [default: None]
You can choose to use --delay or --randdelay. If you use both --delay takes precedence. The --randdelay parameter creates a random delay between a lower and an upper bound that will be added between each insertion event.
The transactions_main.py program knows to use the txntest replica set and the right default port range.
To run the program without transactions you can run it with no arguments:
1$ python transaction_main.py
2using collection: SEATSDB.seats
3using collection: PAYMENTSDB.payments
4using collection: AUDITDB.audit
5Using a fixed delay of 1.0
6
71. Booking seat: '1A'
81. Sleeping: 1.000
91. Paying 330 for seat '1A'
102. Booking seat: '2A'
112. Sleeping: 1.000
122. Paying 450 for seat '2A'
133. Booking seat: '3A'
143. Sleeping: 1.000
153. Paying 490 for seat '3A'
164. Booking seat: '4A'
174. Sleeping: 1.000
The program runs a function called book_seat() which books a seat on a plane by adding documents to three collections. First it adds the seat allocation to the seats_collection, then it adds a payment to the payments_collection, finally it updates an audit count in the audit_collection. (This is a much simplified booking process used purely for illustration).
The default is to run the program without using transactions. To use transactions we have to add the command line flag --usetxns. Run this to test that you are running MongoDB 4.0 and that the correct featureCompatibility is configured (it must be set to 4.0). If you install MongoDB 4.0 over an existing /data directory containing 3.6 databases then featureCompatibility will be set to 3.6 by default and transactions will not be available.
Note: If you get the following error running python transaction_main.py --usetxns that means you are picking up an older version of pymongo (older than 3.7.x) for which there is no multi-document transactions support.
1Traceback (most recent call last):
2 File "transaction_main.py", line 175, in
3 total_delay = total_delay + run_transaction_with_retry( booking_functor, session)
4 File "/Users/jdrumgoole/GIT/pymongo-transactions/transaction_retry.py", line 52, in run_transaction_with_retry
5 with session.start_transaction():
6AttributeError: 'ClientSession' object has no attribute 'start_transaction'

Watching Transactions

To actually see the effect of transactions we need to watch what is happening inside the collections SEATSDB.seats and PAYMENTSDB.payments.
We can do this with watch_transactions.py. This script uses MongoDB Change Streams to see what's happening inside a collection in real-time. We need to run two of these in parallel so it's best to line them up side by side.
Here is the watch_transactions.py program:
1$ python watch_transactions.py -h
2usage: watch_transactions.py [-h] [--host HOST] [--collection COLLECTION]
3
4optional arguments:
5 -h, --help show this help message and exit
6 --host HOST mongodb URI for connecting to server [default:
7 mongodb://localhost:27100/?replicaSet=txntest]
8 --collection COLLECTION
9 Watch [default:
10 PYTHON_TXNS_EXAMPLE.seats_collection]
We need to watch each collection so in two separate terminal windows start the watcher.
Window 1:
1$ python watch_transactions.py --watch seats
2Watching: seats
3...
Window 2:
1$ python watch_transactions.py --watch payments
2Watching: payments
3...

What happens when you run without transactions?

Lets run the code without transactions first. If you examine the transaction_main.py code you will see a function book_seats.
1def book_seat(seats, payments, audit, seat_no, delay_range, session=None):
2 '''
3 Run two inserts in sequence.
4 If session is not None we are in a transaction
5
6 :param seats: seats collection
7 :param payments: payments collection
8 :param seat_no: the number of the seat to be booked (defaults to row A)
9 :param delay_range: A tuple indicating a random delay between two ranges or a single float fixed delay
10 :param session: Session object required by a MongoDB transaction
11 :return: the delay_period for this transaction
12 '''
13 price = random.randrange(200, 500, 10)
14 if type(delay_range) == tuple:
15 delay_period = random.uniform(delay_range[0], delay_range[1])
16 else:
17 delay_period = delay_range
18
19 # Book Seat
20 seat_str = "{}A".format(seat_no)
21 print(count( i, "Booking seat: '{}'".format(seat_str)))
22 seats.insert_one({"flight_no" : "EI178",
23 "seat" : seat_str,
24 "date" : datetime.datetime.utcnow()},
25 session=session)
26 print(count( seat_no, "Sleeping: {:02.3f}".format(delay_period)))
27 #pay for seat
28 time.sleep(delay_period)
29 payments.insert_one({"flight_no" : "EI178",
30 "seat" : seat_str,
31 "date" : datetime.datetime.utcnow(),
32 "price" : price},
33 session=session)
34 audit.update_one({ "audit" : "seats"}, { "$inc" : { "count" : 1}}, upsert=True)
35 print(count(seat_no, "Paying {} for seat '{}'".format(price, seat_str)))
36
37 return delay_period
This program emulates a very simplified airline booking with a seat being allocated and then paid for. These are often separated by a reasonable time frame (e.g. seat allocation vs external credit card validation and anti-fraud check) and we emulate this by inserting a delay. The default is 1 second.
Now with the two watch_transactions.py scripts running for seats_collection and payments_collection we can run transactions_main.py as follows:
1$ python transaction_main.py
The first run is with no transactions enabled.
The bottom window shows transactions_main.py running. On the top left we are watching the inserts to the seats collection. On the top right we are watching inserts to the payments collection.
watching without transactions
We can see that the payments window lags the seats window as the watchers only update when the insert is complete. Thus seats sold cannot be easily reconciled with corresponding payments. If after the third seat has been booked we CTRL-C the program we can see that the program exits before writing the payment. This is reflected in the Change Stream for the payments collection which only shows payments for seat 1A and 2A versus seat allocations for 1A, 2A and 3A.
If we want payments and seats to be instantly reconcilable and consistent we must execute the inserts inside a transaction.

What happens when you run with Transactions?

Now lets run the same system with --usetxns enabled.
1$ python transaction_main.py --usetxns
We run with the exact same setup but now set --usetxns.
watching with transactions
Note now how the change streams are interlocked and are updated in parallel. This is because all the updates only become visible when the transaction is committed. Note how we aborted the third transaction by hitting CTRL-C. Now neither the seat nor the payment appear in the change streams unlike the first example where the seat went through.
This is where transactions shine in world where all or nothing is the watchword. We never want to keeps seats allocated unless they are paid for.

What happens during failure?

In a MongoDB replica set all writes are directed to the Primary node. If the primary node fails or becomes inaccessible (e.g. due to a network partition) writes in flight may fail. In a non-transactional scenario the driver will recover from a single failure and retry the write. In a multi-document transaction we must recover and retry in the event of these kinds of transient failures. This code is encapsulated in transaction_retry.py. We both retry the transaction and retry the commit to handle scenarios where the primary fails within the transaction and/or the commit operation.
1def commit_with_retry(session):
2 while True:
3 try:
4 # Commit uses write concern set at transaction start.
5 session.commit_transaction()
6 print("Transaction committed.")
7 break
8 except (pymongo.errors.ConnectionFailure, pymongo.errors.OperationFailure) as exc:
9 # Can retry commit
10 if exc.has_error_label("UnknownTransactionCommitResult"):
11 print("UnknownTransactionCommitResult, retrying "
12 "commit operation ...")
13 continue
14 else:
15 print("Error during commit ...")
16 raise
17
18def run_transaction_with_retry(functor, session):
19 assert (isinstance(functor, Transaction_Functor))
20 while True:
21 try:
22 with session.start_transaction():
23 result=functor(session) # performs transaction
24 commit_with_retry(session)
25 break
26 except (pymongo.errors.ConnectionFailure, pymongo.errors.OperationFailure) as exc:
27 # If transient error, retry the whole transaction
28 if exc.has_error_label("TransientTransactionError"):
29 print("TransientTransactionError, retrying "
30 "transaction ...")
31 continue
32 else:
33 raise
34
35 return result
In order to observe what happens during elections we can use the script kill_primary.py. This script will start a replica-set and continuously kill the primary.
1$ make kill_primary
2. venv/bin/activate && python kill_primary.py
3no nodes started.
4Current electionTimeoutMillis: 500
51. (Re)starting replica-set
6no nodes started.
71. Getting list of mongod processes
8Process list written to mlaunch.procs
91. Getting replica set status
101. Killing primary node: 31029
111. Sleeping: 1.0
122. (Re)starting replica-set
13launching: "/usr/local/mongodb/bin/mongod" on port 27101
142. Getting list of mongod processes
15Process list written to mlaunch.procs
162. Getting replica set status
172. Killing primary node: 31045
182. Sleeping: 1.0
193. (Re)starting replica-set
20launching: "/usr/local/mongodb/bin/mongod" on port 27102
213. Getting list of mongod processes
22Process list written to mlaunch.procs
233. Getting replica set status
243. Killing primary node: 31137
253. Sleeping: 1.0
kill_primary.py resets electionTimeOutMillis to 500ms from its default of 10000ms (10 seconds). This allows elections to resolve more quickly for the purposes of this test as we are running everything locally.
Once kill_primary.py is running we can start up transactions_main.py again using the --usetxns argument.
1$ make usetxns
2. venv/bin/activate && python transaction_main.py --usetxns
3Forcing collection creation (you can't create collections inside a txn)
4Collections created
5using collection: PYTHON_TXNS_EXAMPLE.seats
6using collection: PYTHON_TXNS_EXAMPLE.payments
7using collection: PYTHON_TXNS_EXAMPLE.audit
8Using a fixed delay of 1.0
9Using transactions
10
111. Booking seat: '1A'
121. Sleeping: 1.000
131. Paying 440 for seat '1A'
14Transaction committed.
152. Booking seat: '2A'
162. Sleeping: 1.000
172. Paying 330 for seat '2A'
18Transaction committed.
193. Booking seat: '3A'
203. Sleeping: 1.000
21TransientTransactionError, retrying transaction ...
223. Booking seat: '3A'
233. Sleeping: 1.000
243. Paying 240 for seat '3A'
25Transaction committed.
264. Booking seat: '4A'
274. Sleeping: 1.000
284. Paying 410 for seat '4A'
29Transaction committed.
305. Booking seat: '5A'
315. Sleeping: 1.000
325. Paying 260 for seat '5A'
33Transaction committed.
346. Booking seat: '6A'
356. Sleeping: 1.000
36TransientTransactionError, retrying transaction ...
376. Booking seat: '6A'
386. Sleeping: 1.000
396. Paying 380 for seat '6A'
40Transaction committed.
41...
As you can see during elections the transaction will be aborted and must be retried. If you look at the transaction_rety.py code you will see how this happens. If a write operation encounters an error it will throw one of the following exceptions:
Within these exceptions there will be a label called TransientTransactionError. This label can be detected using the has_error_label(label) function which is available in pymongo 3.7.x. Transient errors can be recovered from and the retry code in transactions_retry.py has code that retries for both writes and commits (see above).

Conclusion

Multi-document transactions are the final piece of the jigsaw for SQL developers who have been shying away from trying MongoDB. ACID transactions make the programmer's job easier and give teams that are migrating from an existing SQL schema a much more consistent and convenient transition path.
As most migrations involving a move from highly normalised data structures to more natural and flexible nested JSON documents one would expect that the number of required multi-document transactions will be less in a properly constructed MongoDB application. But where multi-document transactions are required programmers can now include them using very similar syntax to SQL.
With ACID transactions in MongoDB 4.0 it can now be the first choice for an even broader range of application use cases.
If you haven't yet set up your free cluster on MongoDB Atlas, now is a great time to do so. You have all the instructions in this blog post.
To try it locally download MongoDB 4.0.

Facebook Icontwitter iconlinkedin icon
Rate this quickstart
star-empty
star-empty
star-empty
star-empty
star-empty
Related
Tutorial

Adding Semantic Caching and Memory to Your RAG Application Using MongoDB and LangChain


Aug 13, 2024 | 16 min read
Tutorial

Spark Up Your MongoDB and BigQuery Using BigQuery Spark Stored Procedures


Aug 12, 2024 | 5 min read
Article

Introducing FARM Stack - FastAPI, React, and MongoDB


Apr 02, 2024 | 7 min read
Tutorial

How to Optimize LLM Applications With Prompt Compression Using LLMLingua and LangChain


Jun 18, 2024 | 13 min read
Table of Contents