*joint with Blake Pollard*

Lately we’ve been thinking about open Markov processes. These are random processes where something can hop randomly from one state to another (that’s the ‘Markov process’ part) but also enter or leave the system (that’s the ‘open’ part).

The ultimate goal is to understand the nonequilibrium thermodynamics of open systems—systems where energy and maybe matter flows in and out. If we could understand this well enough, we could understand in detail how *life* works. That’s a difficult job! But one has to start somewhere, and this is one place to start.

We have a few papers on this subject:

• Blake Pollard, A Second Law for open Markov processes. (Blog article here.)

• John Baez, Brendan Fong and Blake Pollard, A compositional framework for Markov processes. (Blog article here.)

• Blake Pollard, Open Markov processes: A compositional perspective on non-equilibrium steady states in biology. (Blog article here.)

However, right now we just want to show you three closely connected results about how relative entropy changes in open Markov processes.

### Definitions

An **open Markov process** consists of a finite set of **states**, a subset of **boundary states**, and an **infinitesimal stochastic** operator meaning a linear operator with

and

For each state we introduce a **population** We call the resulting function the **population distribution**.

Populations evolve in time according to the **open master equation**:

So, the populations obey a linear differential equation at states that are not in the boundary, but they are specified ‘by the user’ to be chosen functions at the boundary states. The off-diagonal entry for describe the rate at which population transitions from the th to the th state.

A **closed Markov process**, or continuous-time discrete-state Markov chain, is an open Markov process whose boundary is empty. For a closed Markov process, the open master equation becomes the usual **master equation**:

In a closed Markov process the total population is conserved:

This lets us normalize the initial total population to 1 and have it stay equal to 1. If we do this, we can talk about *probabilities* instead of populations. In an open Markov process, population can flow in and out at the boundary states.

For any pair of distinct states is the flow of population from to The **net flux** of population from the th state to the th state is the flow from to minus the flow from to :

A **steady state** is a solution of the open master equation that does not change with time. A steady state for a closed Markov process is typically called an **equilibrium**. So, an equilibrium obeys the master equation at all states, while for a steady state this may not be true at the boundary states. The idea is that population can flow in or out at the boundary states.

We say an equilibrium of a Markov process is **detailed balanced** if all the net fluxes vanish:

or in other words:

Given two population distributions we can define the **relative entropy**

When is a detailed balanced equilibrium solution of the master equation, the relative entropy can be seen as the ‘free energy’ of For a precise statement, see Section 4 of Relative entropy in biological systems.

The Second Law of Thermodynamics implies that the free energy of a closed system tends to decrease with time, so for *closed* Markov processes we expect to be nonincreasing. And this is true! But for *open* Markov processes, free energy can flow in from outside. This is just one of several nice results about how relative entropy changes with time.

### Results

**Theorem 1.** Consider an open Markov process with as its set of states and as the set of boundary states. Suppose and obey the open master equation, and let the quantities

measure how much the time derivatives of and fail to obey the master equation. Then we have

This result separates the change in relative entropy change into two parts: an ‘internal’ part and a ‘boundary’ part.

It turns out the ‘internal’ part is always less than or equal to zero. So, from Theorem 1 we can deduce a version of the Second Law of Thermodynamics for open Markov processes:

**Theorem 2.** Given the conditions of Theorem 1, we have

Intuitively, this says that free energy can only increase if it comes in from the boundary!

There is another nice result that holds when is an equilibrium solution of the master equation. This idea seems to go back to Schnakenberg:

**Theorem 3.** Given the conditions of Theorem 1, suppose also that is an equilibrium solution of the master equation. Then we have

where

is the **net flux** from to while

is the conjugate **thermodynamic force**.

The flux has a nice meaning: it’s the net flow of population from to The thermodynamic force is a bit subtler, but this theorem reveals its meaning: it says how much the population *wants* to flow from to

More precisely, up to that factor of the thermodynamic force says how much free energy loss is caused by net flux from to There’s a nice analogy here to water losing potential energy as it flows downhill due to the force of gravity.

### Proofs

**Proof of Theorem 1.** We begin by taking the time derivative of the relative information:

We can separate this into a sum over states for which the time derivatives of and are given by the master equation, and boundary states for which they are not:

For boundary states we have

and similarly for the time derivative of We thus obtain

To evaluate the first sum, recall that

so

Thus, we have

We can rewrite this as

Since is infinitesimal stochastic we have so the first term drops out, and we are left with

as desired. █

**Proof of Theorem 2.** Thanks to Theorem 1, to prove

it suffices to show that

or equivalently (recalling the proof of Theorem 1):

The last two terms on the left hand side cancel when Thus, if we break the sum into an part and an part, the left side becomes

Next we can use the infinitesimal stochastic property of to write as the sum of over not equal to obtaining

Since when and for all we conclude that this quantity is █

**Proof of Theorem 3.** Now suppose also that is an equilibrium solution of the master equation. Then for all states so by Theorem 1 we need to show

We also have so the second

term in the sum at left vanishes, and it suffices to show

By definition we have

This in turn equals

and we can switch the dummy indices in the second sum, obtaining

or simply

But this is

and the first term vanishes because is infinitesimal stochastic: We thus have

as desired. █