joint with Blake Pollard
Lately we’ve been thinking about open Markov processes. These are random processes where something can hop randomly from one state to another (that’s the ‘Markov process’ part) but also enter or leave the system (that’s the ‘open’ part).
The ultimate goal is to understand the nonequilibrium thermodynamics of open systems—systems where energy and maybe matter flows in and out. If we could understand this well enough, we could understand in detail how life works. That’s a difficult job! But one has to start somewhere, and this is one place to start.
We have a few papers on this subject:
• Blake Pollard, Open Markov processes: A compositional perspective on non-equilibrium steady states in biology. (Blog article here.)
However, right now we just want to show you three closely connected results about how relative entropy changes in open Markov processes.
An open Markov process consists of a finite set of states, a subset of boundary states, and an infinitesimal stochastic operator meaning a linear operator with
For each state we introduce a population We call the resulting function the population distribution.
Populations evolve in time according to the open master equation:
So, the populations obey a linear differential equation at states that are not in the boundary, but they are specified ‘by the user’ to be chosen functions at the boundary states. The off-diagonal entry for describe the rate at which population transitions from the th to the th state.
A closed Markov process, or continuous-time discrete-state Markov chain, is an open Markov process whose boundary is empty. For a closed Markov process, the open master equation becomes the usual master equation:
In a closed Markov process the total population is conserved:
This lets us normalize the initial total population to 1 and have it stay equal to 1. If we do this, we can talk about probabilities instead of populations. In an open Markov process, population can flow in and out at the boundary states.
For any pair of distinct states is the flow of population from to The net flux of population from the th state to the th state is the flow from to minus the flow from to :
A steady state is a solution of the open master equation that does not change with time. A steady state for a closed Markov process is typically called an equilibrium. So, an equilibrium obeys the master equation at all states, while for a steady state this may not be true at the boundary states. The idea is that population can flow in or out at the boundary states.
We say an equilibrium of a Markov process is detailed balanced if all the net fluxes vanish:
or in other words:
Given two population distributions we can define the relative entropy
When is a detailed balanced equilibrium solution of the master equation, the relative entropy can be seen as the ‘free energy’ of For a precise statement, see Section 4 of Relative entropy in biological systems.
The Second Law of Thermodynamics implies that the free energy of a closed system tends to decrease with time, so for closed Markov processes we expect to be nonincreasing. And this is true! But for open Markov processes, free energy can flow in from outside. This is just one of several nice results about how relative entropy changes with time.
Theorem 1. Consider an open Markov process with as its set of states and as the set of boundary states. Suppose and obey the open master equation, and let the quantities
measure how much the time derivatives of and fail to obey the master equation. Then we have
This result separates the change in relative entropy change into two parts: an ‘internal’ part and a ‘boundary’ part.
It turns out the ‘internal’ part is always less than or equal to zero. So, from Theorem 1 we can deduce a version of the Second Law of Thermodynamics for open Markov processes:
Theorem 2. Given the conditions of Theorem 1, we have
Intuitively, this says that free energy can only increase if it comes in from the boundary!
There is another nice result that holds when is an equilibrium solution of the master equation. This idea seems to go back to Schnakenberg:
Theorem 3. Given the conditions of Theorem 1, suppose also that is an equilibrium solution of the master equation. Then we have
is the net flux from to while
is the conjugate thermodynamic force.
The flux has a nice meaning: it’s the net flow of population from to The thermodynamic force is a bit subtler, but this theorem reveals its meaning: it says how much the population wants to flow from to
More precisely, up to that factor of the thermodynamic force says how much free energy loss is caused by net flux from to There’s a nice analogy here to water losing potential energy as it flows downhill due to the force of gravity.
Proof of Theorem 1. We begin by taking the time derivative of the relative information:
We can separate this into a sum over states for which the time derivatives of and are given by the master equation, and boundary states for which they are not:
For boundary states we have
and similarly for the time derivative of We thus obtain
To evaluate the first sum, recall that
Thus, we have
We can rewrite this as
Since is infinitesimal stochastic we have so the first term drops out, and we are left with
as desired. █
Proof of Theorem 2. Thanks to Theorem 1, to prove
it suffices to show that
or equivalently (recalling the proof of Theorem 1):
The last two terms on the left hand side cancel when Thus, if we break the sum into an part and an part, the left side becomes
Next we can use the infinitesimal stochastic property of to write as the sum of over not equal to obtaining
Since when and for all we conclude that this quantity is █
Proof of Theorem 3. Now suppose also that is an equilibrium solution of the master equation. Then for all states so by Theorem 1 we need to show
We also have so the second
term in the sum at left vanishes, and it suffices to show
By definition we have
This in turn equals
and we can switch the dummy indices in the second sum, obtaining
But this is
and the first term vanishes because is infinitesimal stochastic: We thus have
as desired. █