Markov Decision Processes in Finance and Dynamic Options. Manfred Schäl Inst. Angew. Math., Univ. Bonn, Wegelerstr.6, D − 53115 Bonn Draft from May 24, 2000 Dedicated to Professor Dr. Karl Hinderer Abstract In this paper a discrete-time Markovian model for a financial market is chosen. The

fundamental theorem of asset pricing relates the existence of a martingale measure to the no-arbitrage condition. It is explained how to prove the theorem by stochastic dynamic programming via portfolio optimization. The approach singles out certain martingale measures with additional interesting properties. Furthermore, it is shown how to use dynamic programming to study the smallest initial wealth x∗ that allows for super-hedging a contingent claim by some dynamic portfolio. There, a joint property of the set of policies in a Markov decision model and the set of martingale measures is exploited. The approach extends to dynamic options which are introduced here and are generalizations of American options.

1 Introduction and Summary In discrete time n = 0, 1, ..., N a financial market is studied which is free of arbitrage opportunities but incomplete. In the market 1 + d assets can be traded. One of them with price process {B_{n}, 0 ≤ n ≤ N} is called the bond or savings account and is assumed to be nonrisky. The other d assets are called stocks and are described by the d-dimensional price process {S_{n}, 0 ≤ n ≤ N}. An investor is considered whose attitude towards risk is specified in terms of a utility function U. A dynamic portfolio is specified by a policy ϕ. The investor’s objective is to maximize the expected utility of the discounted terminal wealth X_{N}^{ϕ}(x) when starting with an initial wealth x . Utility optimization is now a classical subject (Bertsekas [1], Hakansson [16]). The present paper is intended as a bridge between Markov decision processes (MDPs) and modern concepts of finance. It is assumed the reader knows some basic facts about MDPs, but knowledge about Mathematical Finance is not needed. In particular, we here are interested in using the no-arbitrage condition for the construction of a particular martingale measure by use of the optimal policy ϕ∗. A condition close to the no-arbitrage condition was already used by Hakansson [16]. Martingale measures are used for option pricing. The paper makes use of an approach how to base option pricing on the optimal solution ϕ∗ to the portfolio optimization problem. This approach is also explained by Davis [4]. In a one-period model, the construction of a martingale measure by use of an optimization problem is easily explained. Let us assume d = 1. If the action (portfolio) a ∈ R is chosen then the discounted terminal wealth of the portfolio has the form x + a · R where R can be interpreted as a return. If a = a∗ is optimal then one has ∂E[U(x + a · R)]/∂a = 0 = E[U′(x + a · R) · R] for a = a∗. Upon defining a new measure Q by dQ = const · U′(x + a ∗ ·R)dP , one obtains ∫ R dQ = E_{Q}[R] = 0 which is the martingale property. Since the martingale property is a local one both in time and in space, martingale measures can be constructed by local optimization problems (see Rogers 1994) whereas we will apply global (dynamic) optimization. This has the advantage that the resulting martingale measures have interesting interpretations. The case where the utility function U is only defined for positive values is treated in [38], [39]. A second interesting problem concerns values x for the initial wealth that allow for superhedging discounted contingent claims X̆ by some policy ϕ, i.e. X_{N}^{ϕ}(x) ≥ X̆. The smallest value x∗ coincides with the maximal expectation of X̆ under (equivalent) martingale measures. The proof can make use of dynamic programming by exploiting an analogy between the set of all policies in a stochastic dynamic programming model and the set of martingale measures. A similar problem can be considered for American options where an optimal stopping time has to be chosen. It is well-known that an optimal stopping problem can be considered as a special stochastic dynamic programming problem. Therefore it is natural from the point of view of Markov decision theory to generalize the concept of an American option. This is done in the present paper and the generalization is called a dynamic option which has interesting applications.

2 The financial market On the market an investor can observe the prices of 1 + d securities at the dates n = 0, 1, ..., N where N is the time horizon. One of the securities is a bond (or savings account) with interest rates r_{n}, 1 ≤ n ≤ N. It is essential for the theory that the interest rates for borrowing and lending are assumed to be the same. The bond price process is defined by (2.1) B_{n} := (1 + r_{1}) · ... · (1 + r_{n}) , 0 ≤ n ≤ N, where B_{0} = 1. Here we assume that {B_{n}} is a deterministic process. If {B_{n}} is given as the initial term structure, then the interest rates r_{n} can be computed by (2.1). We will have r_{n} ≥ 0, but we only need that 1 + r_{n} > 0. The other d securities are called stocks. The evolution of the stock prices will be modeled by a d-dimensional stochastic process {S_{n}, n = 0, 1, ..., N} where S_{0} is deterministic. There the components S_{n}^{k} of S_{n}, 1 ≤ k ≤ d, are assumed to be positive. One may also think of a foreign exchange market where S_{n}^{k} is the exchange rate for a foreign currency. Besides the savings account, the investor has d accounts for different foreign exchanges. The value of these accounts as well as that of the savings account may be negative which can be interpreted as a loan. The information about the market at time n including the observed stock prices will be represented by a Markov chain (I_{n}, 0 ≤ n ≤ N) on some probability space (Ω, P) where I_{n} takes on values in some space E_{n}, 0 ≤ n ≤ N, where I_{0} = i_{0} is a given constant and hence E_{0} = {i_{0}}. In order to avoid integrability and measurability problems we assume here: (2.2) E_{n} is finite, 1 ≤ n ≤ N, hence (w.l.o.g.)Ω is finite and P [{ω}] > 0, ω ∈ Ω. For any vector-valued process {Z_{n}}, we define the backward increment by ΔZ_{n} := Z_{n} − Z_{n−1}. Further, we write ζ^{⊤} · ξ for the inner product of ξ, ζ ∈ R^{d}. The representation often becomes easier (see (2.6) below) if one considers the discounted stock price process S̆_{n} = (S̆_{n}^{1}, ..., S̆_{n}^{d}) defined by (2.3) S̆_{n}^{k} := S_{n}^{k}/ B_{n} , k = 1, ..., d, n = 0, ..., N. The relative risk process {R_{n} = (R_{n}^{1}, ..., R_{n}^{d}) , 1 ≤ n ≤ N} (Karatzas & Kou [22]) is defined by (2.4) 1 + R_{n}^{k} := 1 + ΔS̆_{n}^{k}/S̆_{n−1}^{k} = \frac{1}{1+r_{n}} \{1 + ΔS_{n}^{k}/ S_{n−1}^{k}\} where {ΔS_{n}^{k}/S_{n−1}^{k}, 1 ≤ n ≤ N} is the return process corresponding to {S_{n}^{k}, 0 ≤ n ≤ N} (Pliska [31]§3.2). Then we get (2.5) S̆_{n}^{k} = S̆_{n−1}^{k} · (1 + R_{n}^{k}) = S_{0}^{k} · (1 + R_{1}^{k}) · ... · (1 + R_{n}^{k}) .

Since the investor can observe S_{n} and R_{n}, it is natural to assume that S_{n} and R_{n} are known if the history I_{0}, ..., I_{n} is known [i.e. that S_{n} and R_{n} are adapted in the sense explained in section 3]. In order to get a Markovian structure, we assume that R_{n} is a function R_{n} = ρ_{n}(I_{n−1}, I_{n}) for some function ρ_{n} on E_{n−1} × E_{n}. Important examples are I_{n} = S_{n} or I_{n} = R_{n}. The latter example I_{n} = R_{n} is sometimes convenient for a model where R_{1}, ..., R_{N} are independent random variables as in many papers on portfolio optimization. An example for independent random variables R_{1}, ..., R_{N} is the so-called Binomial model; there one has d = 1 and r_{n} = r (independent of n). It is defined by two numbers δ and u such that 0 < δ < 1 + r < u and S_{n} = (1 + r) · (1 + R_{n}) · S_{n−1} is either equal to u · S_{n−1} or δ · S_{n−1}. If we choose I_{n} = R_{n}, then we get E_{n} = {ι/(1 + r) − 1; ι = δ, u} (which is independent of n). However, sometimes [when considering so-called contingent claims X = f(S_{N}) in section 3] one should choose I_{n} = S_{n} and then E_{n} = {u·i, i ∈ E_{n−1}}∪{δ ·i, i ∈ E_{n−1}} . This example shows that it is useful to let E_{n} depend on n. At each time n, the investor will obtain the market information I_{n} = i and observe the discounted value x_{n} of his portfolio. Then he will decide about the amount ϕ_{n}^{k} invested in stock k during (n, n + 1], i.e. , ϕ_{n}^{k}/S_{n}^{k} denotes the number of shares the investor holds during (n, n+1]. The decision may depend on the present information (i_{n}, x_{n}). A dynamic portfolio is here described by a (Markov) policy ϕ which is given by a sequence of functions ϕ = {ϕ_{n}, 0 ≤ n < N} where ϕ_{n} is a mapping from E_{n} × R to R^{d}. In particular, one allows for negative amounts ϕ_{n}^{k}, i.e. one allows for short selling of stocks. In the case of a foreign exchange market one can think of a negative ϕ_{n}^{k} as a loan. Given the initial wealth x, the amount η_{n} invested in the bond in [n, n + 1) is then specified by ϕ according to the following budget equation where we write 1 ∼ for the d-dimensional vector with every

tcol

component equal to 1.

(2.6) η_{0} + ϕ_{0}^{⊤} · 1 ∼ = x, tcol

η_{n} + ϕ_{n}^{⊤} · 1 ∼ = η_{n−1} · (1 + r_{n}) + ∑_{k=1}^{d} ϕ_{n−1}^{k} · S_{n}^{k}/S_{n−1}^{k} tcol

= (1 + r_{n}) · [η_{n−1} + ϕ_{n−1}^{⊤} · (1 ∼ + R_{n})], 1 ≤ n < N. tcol

The investor can choose any new portfolio ϕ_{n} at time n for the stocks. This decision is then compensated by the savings account. In particular, this means that no funds are added to or withdrawn from the wealth of the portfolio at any time. All changes in wealth are due to capital gains (appreciation of stocks and interest from the bond). Then the policy is called self-financing. Again it is convenient to work with the discounted wealth of the portfolio in place of the wealth itself. The discounted wealth process {X_{n}^{ϕ}(x)} is given through

(2.7) X_{n}^{ϕ}(x) := [η_{n} + ϕ_{n}^{⊤} · 1 ∼]/B_{n} = [η_{n−1} + ϕ_{n−1}^{⊤} · (1 ∼ + R_{n})] / B_{n−1} , 0 ≤ n < N, tcol tcol

X_{N}^{ϕ}(x) := [η_{N−1} + ϕ_{N−1}^{⊤} · (1 ∼ + R_{N})] / B_{N−1} . tcol

At time N there is no rebalancing of the portfolio. Obviously we have (2.8) X_{n}^{ϕ}(x) = X_{n−1}^{ϕ}(x) + ϕ_{n−1}^{⊤} · R_{n} . It is also usual to describe a portfolio by the numbers ξ_{n}^{k} := ϕ_{n}^{k}/S_{n}^{k} of shares the investor holds. In the case where the wealth is positive it can be useful to describe a portfolio by the proportions π_{n}^{k} := ϕ_{n}^{k}/X_{n}^{ϕ}(x) of the wealth which are invested in the stocks. But here it will be convenient to use the description of a portfolio by the amounts ϕ_{n}^{k}. In the present approach we have E_{n} × R as state space of the underlying Markov decision process and we tried to explain why it is useful to let E_{n} depend on n. It is well-known that one can transform the present non-stationary Markovian model to a stationary model by including the time-parameter in the state (Feinberg [9]); but we will stick to the nonstationary setting. The action space is R^{d} and all actions a ∈ R^{d} are admissible at each time and state. Given the initial wealth x, the history I_{0}, ..., I_{n}, and the decisions ϕ_{0}, ..., ϕ_{n−1}, we know the discounted value X_{n}^{ϕ}(x) = x_{n} of the portfolio. Therefore, one could dispense with x_{n} as part of the state. However, the present choice (i_{n}, x_{n}) of the state makes the model Markovian. In the case (mostly considered in the literature) when R_{n} = I_{n} and the R_{1}, ..., R_{N} are independent, one can even dispense with i_{n} as part of the state and just choose x_{n} as the state at time n. As underlying basic process we choose the process {I_{n}} representing the informations about the market. There are transition probabilities p_{ij}(n), i ∈ E_{n−1}, j ∈ E_{n}, where p_{ij}(n) specifies the conditional probability P [I_{n+1} = j|I_{n} = i] given the present value I_{n} = i [and the past (i_{0}, ..., i_{n−1})]. It is a particular feature that the probability measure P describing the dynamics does not depend on the policy. This will be important when considering other artificial probability measures Q on Ω in the next section. As explained above, the state process for the Markov decision process is {(I_{n}, X_{n}^{ϕ}(x))}. The distribution of the state process does depend on the policy as usual. Therefore, the measure P describing the dynamics of {I_{n}} is here more appropriate than the policy depending (strategic) measure describing the dynamics of {(I_{n}, X_{n}^{ϕ}(x))} on the canonical space of trajectories of length N. If {I_{n}} is a non-Markovian process, it can be transformed to the present non-stationary Markovian model by choosing the history space E_{0} × ... × E_{n} as new space Ẽ_{n}.

3 No-arbitrage and martingale measures In order to use the important concept of martingales w.r.t. to the given information structure , we have to use conditional expectations ζ(i_{1}, ..., i_{n}) := E[Z|I_{1} = i_{1}, ..., I_{n} = i_{n}] defined on E_{1} × ... × E_{n} and E[Z|σ(I_{1}, ..., I_{n})] defined on Ω for a random variable Z. As usual, we use the convention that E[Z|σ(I_{1}, ..., I_{n})](ω) := ζ(I_{1}(ω), ..., I_{n}(ω)), ω ∈ Ω. Then a real-valued stochastic process {Z_{n}, 0 ≤ n ≤ N} is a martingale if Z_{n} is adapted to the information structure, i.e., Z_{n} is a function of (I_{1}, ..., I_{n}) where Z_{0} is a constant and if

E[ΔZ_{n}|σ(I_{1}, ..., I_{n−1})] = 0, 1 ≤ n ≤ N. It is necessary to consider further probability measures Q on Ω which are equivalent (to the given physical probability measure P); i.e., Q[{ω}] > 0 , ω ∈ Ω. We write E_{Q}[Z] = ∫ Z dQ for the expectation of the random variable Z under Q whereas E[Z] is the expectation of the random variable Z under P as usual. Definition 1. Q is called a martingale measure iff {S̆_{n}} forms a martingale under Q, i.e., (3.1a) E_{Q}[ΔS̆_{n}^{k}|σ(I_{1}, ..., I_{n−1})] = 0 , 1 ≤ k ≤ d, 1 ≤ n ≤ N. Obviously (3.1a) is equivalent to (3.1b) E_{Q}[R_{n}^{k}|σ(I_{1}, ..., I_{n−1})] = 0 , 1 ≤ k ≤ d, 1 ≤ n ≤ N. Then we set (3.2) Q := {Q; Q is an equivalent martingale measure}. Equivalent martingale measures are used for the valuation of contingent claims (Harrison & Kreps [16]). A contingent claim is a random variable X that represents the time N payoff from a ’seller’ to a ’buyer’. In most instances the random variable X can be taken to be some function of an underlying stock price, and so contingent claims are examples of what are called derivative securities. In the present framework, it is natural to assume that X is a function of I_{N} and we recall that we may choose I_{n} = S_{n} as an example. A typical example is given by a European call option. At time n = 0 the buyer signs a contract which gives him the option to buy, at a specified time N (called the maturity date or expiration date), one share of stock 1 at a specified price χ, called the exercise price. At maturity, if the price S_{N}^{1} of stock 1 is below the exercise price, the contract is worthless to the buyer; on the other hand, if S_{N}^{1} > χ, the buyer can exercise his option (i.e., buy one share at the preassigned price χ) and then sell the share immediately in the market for S_{N}^{1}. This contract is thus equivalent to a payment of X = (S_{N}^{1} − χ)^{+} (the contingent claim) at maturity. Given the price dynamics of the securities, one tries to determine the prices of such a contingent claim X. A classical answer in the sense of Huygens and Bernoulli to the question of a fair price π for the option would rely on the concept of a fair game and would then be π = E[X/B_{N}]. There is a different answer in the present situation where the seller can hedge X by investing in the stocks For example if S_{N}^{1} is high, then the buyer’s and the seller’s gain is high. A contingent claim X is said to be attainable if there exists a policy ϕ and some x ∈ R such that X̆ = X_{N}^{ϕ}(x) for X̆ := X/B_{N}. The corresponding policy ϕ is said to replicate X, since the (nondiscounted) wealth B_{N} · X_{N}^{ϕ}(x) of the dynamic portfolio ϕ at time N replicates X. This is the case if χ = 0, i.e. X = S_{N}^{1}, in the example above; then one can choose x = S_{0}^{1} and ϕ_{n}^{1} ≡ S_{n}^{1}, ϕ_{n}^{k} ≡ 0, k ≠ 1, for all n. It is clear that for attainable contingent claims with X̆ = X_{N}^{ϕ}(x) a fair price is x. The buyer and the seller could instead have invested the wealth x in such a way as to replicate the payoff of the contingent claim. Now from the martingale property one immediately gets (3.3) E_{Q}[X_{N}^{ϕ}(x)] = x for Q ∈ Q. Since a price system should be a linear functional on the space of all contingent claims X, i.e. of all random variables X on Ω (Harrison & Kreps [16]), it becomes clear that (3.4) π_{Q}(X) := E_{Q}[X̆] for Q ∈ Q is a candidate for a fair price. The market is said to be complete if each contingent claim is attainable. An example is the Binomial model for the case d = 1. For complete markets option pricing is no problem. However, there are only very few examples of complete discrete-time markets (Harrison & Pliska [17], Jacod & Shiryaev [20]). In continuous time, the Black-Scholes model is an important example of a complete market. The question of the existence of some Q ∈ Q is strongly connected with the following no-arbitrage condition: (NA) for 1 ≤ n ≤ N and any portfolio ϕ_{n−1} depending on (I_{1}, ..., I_{n−1}) :

ϕ_{n−1}^{⊤} · R_{n} ≥ 0 implies ϕ_{n−1}^{⊤} · R_{n} = 0 . There (NA) means that if there is a chance that ϕ_{n−1}^{⊤} · R_{n} > 0 , then there is also a chance that ϕ_{n−1}^{⊤} ·R_{n} < 0. A portfolio ϕ_{n−1} with “ϕ_{n−1}^{⊤} ·R_{n} ≥ 0” and “ϕ_{n−1}^{⊤} ·R_{n} > 0 for some ω” is called an arbitrage opportunity. This definition of arbitrage is standard in the literature where we here need not care about null sets. Such an arbitrage opportunity represents a riskless source of generating profit, strictly greater than the profit from the bond. In order to present (NA) in full generality, we used a non-Markovian dynamic portfolio ϕ. Now we can present one of the most important results for financial markets:

Fundamental Theorem of Asset Pricing. There exists an equivalent martingale measure, i.e., Q is not empty, if and only if the no-arbitrage condition (NA) holds.

Proofs for general spaces Ω can be found in Dalang et al [3], Schachermayer [35], Rogers [32], Jacod & Shiryaev [20]. It is easy to see that (NA) is necessary for the existence of some Q ∈ Q; therefore we can give the proof here: Let be Q ∈ Q. From ϕ_{n−1}^{⊤} · R_{n} ≥ 0 we conclude that ϕ_{n−1}^{⊤} · R_{n} = 0 Q − a.s. since E_{Q}[ϕ_{n−1}^{⊤} · R_{n}] = 0. As Q is equivalent we have ϕ_{n−1}^{⊤} · R_{n} = 0 everywhere. In this paper it will be proved in §6 by use of dynamic programming that (NA) is also sufficient for the existence of some Q ∈ Q. Since the martingale property is a local one both in time and in space, martingale measures can also be constructed by local optimization problems. This was done by Rogers [32] whereas we will apply global (dynamic) optimization. From now on we use the following assumption: Assumption 1. (NA) holds. Since in discrete time the assumption of completeness is a severe restriction, we are forced to consider general incomplete markets. From the so-called second fundamental theorem of asset pricing it is known (Harrison & Pliska [17], Jacod & Shiryaev [20]) that in incomplete markets with the (NA)-condition one has several choices of equivalent martingale measures (from the convex set Q). Thus in incomplete markets, no preference independent pricing of contingent claims is possible. The approach of this paper, which makes use of dynamic programming, has another interesting consequence about identifying a certain price. There option pricing is based on an optimal solution to the portfolio optimization problem. This approach is also explained by Davis [4]. Suppose a utility function U on R × E_{N} is given where U(x, i) is strictly concave and differentiable in x with partial derivative U′(x, i). The investor’s objective is to maximize the expected utility of terminal wealth and the maximum utility is given by (3.5) V_{0}(x) = sup_{ϕ} E[U(X_{N}^{ϕ}(x), I_{N})] = E[U(X_{N}^{ϕ∗}(x), I_{N})] where ϕ∗ is a solution to the optimization problem. In §6 it will be explained that it may be useful to let the utility depend on I_{N}. Consider a contingent claim X made available for trading with purchase price π. One can ask the question whether the maximum utility in (3.5) can be increased by the purchase (or short-selling) of an option described by X. In order to find a fair price π̂ for X one can use the following argument: π̂ is a fair price for the contingent claim if diverting a little of the investor’s initital wealth into the option at time zero has a neutral effect on the investor’s achievable utility. For N = 1 this concept was also explained by Merton [27]§7.7). More precisely, consider the discounted terminal wealth at time N of an investor who follows the dynamic portfolio ϕ∗ which is optimal for his initial wealth x and who then diverts the amount ξ ∈ R to purchase ξ/π shares of the option with price π and discounted contingent claim X̆ = X/B_{N} :

X_{N}^{ϕ∗}(x − ξ) + {ξ \atop π} · X̆ = x − ξ + X_{N}^{ϕ∗}(0) + {ξ \atop π} · X̆ = X_{N}^{ϕ∗}(x) + ξ · \{{1 \atop π}X̆ − 1\} . tcol tcol tcol

Then the expected utility is

(3.6) g(ϕ∗, ξ, π, x) = E[U(X_{N}^{ϕ∗}(x) + ξ · \{{1 \atop π}X̆ − 1\}, I_{N})] tcol

where sup_{ϕ} g(ϕ, 0, π, x) = V_{0}(x) = g(ϕ∗, 0, π, x). Then one obtains

(3.7) ∂ ; − ; ∂− ξg(ϕ∗, ξ, π, x)|_{ξ=0} = E[U′(X_{N}^{ϕ∗}(x), I_{N}) · \{{1 \atop π}X̆ − 1\}] . tcol tcol tcol tcol

Further, g(ϕ∗, ξ, π, x) is concave in ξ and strictly concave except for the less interesting case where X̆ agrees with π and hence X is attainable.

Now choose π = π̂ such that ∂ ; − ; ∂− ξg(ϕ∗, ξ, π̂, x)|_{ξ=0} = 0 and g(ϕ∗, ξ, π̂, x) is hence maximal

tcol tcol

tcol

for ξ = 0. Thus π̂ is not too high otherwise the investor would like to sell short the option; and π̂ is not too low otherwise the investor would like to purchase the option. This leads to the formula (3.8) π̂ = E[U′(X_{N}^{ϕ∗}(x), I_{N}) · X̆] / E[U′(X_{T}^{ϕ∗}(x), I_{N})] = ∫ X̆dQ_{x}^{U} . The relation (3.8) leads to the interesting result that in our situation the option pricing formula of Davis is given by the martingale measure Q_{x}^{U} which is constructed in §5. A similar result was found by Karatzas & Kou [22] Theorem 7.4, who studied a diffusion model extensively. The case where U is defined only for positive values is treated in [38], [39]. Again consider an option with discounted contingent claim X̆ := X/B_{N} . In the second part of the paper we are interested in (3.9) x∗ := inf{x ∈ R; X_{N}^{ϕ}(x) ≥ X̆ for some ϕ}. Thus we look for values x of the initial wealth that allow for super-hedging X by some dynamic portfolio ϕ. Then a duality result holds: The smallest value x∗ will be shown to coincide with the maximal expectation of X̆ under equivalent martingale measures. Theorem 2. x∗ = sup_{Q∈Q} E_{Q}[X̆]. The quantity x∗ is called upper price, hedging price, upper bound for the fair price, selling price, or arbitrage upper bound. It is known (see e.g. [37] 1.16) that E_{Q}[X̆] < x∗ for each Q ∈ Q unless X is attainable. In view of Theorem 2, a price π = E_{Q}[X̆] (for some Q ∈ Q) thus offers no arbitrage opportunity to the seller in the sense that X_{N}^{ϕ}(π) ≥ X̆ , X_{N}^{ϕ}(π) ≠ X̆ for some ϕ. A quantity similar to x∗ will also be considered for American options (Karatzas [21]) where an optimal stopping time can be chosen by the buyer. Moreover, a generalization of American options is introduced here which is called dynamic option. This is a natural generalization from the point of stochastic dynamic programming where it is known how to imbed an optimal stopping problem in a Markov decision problem.

4 The one-period model In this section we will study the case N = 1. We write R := R_{1} and (4.1) ∑ := {R(ω); ω ∈ Ω} for the support of R. Furthermore, L is the smallest linear space in R^{d} containing ∑, i.e. L is the smallest linear space L in R^{d} such that P [R ∈ L] = 1. Then it is easy to show that (NA) is equivalent to : (NA)_{1} for all a ∈ L \ {0} : P [a^{⊤} · R < 0] > 0 . We start from a utility function U(x, i) depending on two variables and we use the following properties: Assumption 3. U ∈ F_{N} where

F_{n} := {f : R × E_{n} ↦ R such that x ↦ f(x, i) is strictly concave and differentiable with derivative f′(x, i); furthermore, f′(−∞, i) > 0 and f′(+∞, i) ≤ 0}. ¿From the concavity we know that U′(x, i) is decreasing in x. In a one-period model, a policy ϕ specifies just one portfolio a ∈ R^{d}. We obtain from (2.6) and (2.7) that then (4.2) X_{1}^{ϕ}(x) := x + a^{⊤} · R .

We want to maximize the expected utility: (4.3) v(x, a) := E[U(x + a^{⊤} · R, I_{1})] for a ∈ R^{d};

V (x) := sup_{a∈R^{d}} v(x, a) for x ∈ R^{d}. Lemma 4. (a) If Γ denotes the orthogonal projection on L , then a^{⊤} · R = (Γa)^{⊤} · R;

(b) (x, a) ↦ v(x, a) is continuous;

(c) for each x ∈ R, a ↦ v(x, a) attains the maximum on R^{d} where the maximum point can be chosen in L . Proof. The parts (a) and (b) are obvious. For part (c) one can use the methods of Leland [26], Bertsekas [1] or Rogers [32]. By (a) we can restrict attention to a ∈ L . We have

1 ; − ; λ · [v(x, λ · a) − v(x, 0)] tcol

= E[ D(λ, a^{⊤} · R, x, I_{1}) · 1_{{a}⊤_{·R>0}}] + E[D(λ, a^{⊤} · R, x, I_{1}) · 1_{{a}⊤_{·R<0}}].

where D(λ, y, x, i) := 1 ; − ; λ · {U(x + λ y, i) − U(x, i)} is decreasing in λ both for y > 0 and for

tcol

y < 0 because of the concavity of U(·, i). Further D(∞, y, x, i) := lim_{λ ∞} D(λ, y, x, i) ≤ 0 for y > 0 and D(∞, y, x, i) < 0 for y > 0 by our assumption on U. Now, we get by use of the monotone convergence theorem:

lim_{λ ∞} 1 ; − ; λ · [v(x, λ · a) − v(x, 0)] tcol

= E[ D(∞, a^{⊤} · R, x, I_{1}) · 1_{{a}⊤_{·R>0}}] + E[D(∞, a^{⊤} · R, x, I_{1}) · 1_{{a}⊤_{·R<0}}]. By (NA) we have for a ∈ L \ {0} : P [a^{⊤} · R < 0] > 0 and thus:

lim_{λ ∞} 1 ; − ; λ · [v(x, λ · a) − v(x, 0)] < 0 for a ∈ L \ {0}. tcol

In particular, we have lim_{λ ∞} v(x, λ · a) = −∞ for a ∈ L \ {0}. Now the result follows (see Rockafellar [33] Theorems 27.1, 27.3, Bertsekas [1] Proposition 1, Rogers [32] Proposition 2.2). [] Lemma 5. (a) v(x, a) is strictly concave in x for fixed a;

(b) V (x) is strictly concave in x and the maximum point of a ↦ v(x, a) in L is unique. Proof. Part (a) is obvious. For the proof of (b) choose x, x̃ ∈ R and λ, λ̃ > 0 such that λ+λ̃ = 1. Further choose a ∈ L such that V (x) = v(x, a) and similarly ã for x̃. We consider the cases (i) x̃ > x and (ii) x̃ = x and a ≠ ã. Then P [x+a^{⊤}·R ≠ x̃+ã^{⊤}·R]] > 0 ; otherwise (a − ã)^{⊤} · R = x̃ − x which contradicts (NA) in case (i) and contradicts a − ã ∈ L \ {0} in case (ii). Now we obtain from the strict concavity of U(·, i) : λ · V (x) + λ̃ · V (x̃) = λ · v(x, a) + λ̃ · v(x̃, ã) < v(λ · x + λ̃ · x̃ , λ · a + λ̃ · ã) ≤ V (λ · x + λ̃ · x̃) . In case (ii) we have a contradiction. [] Lemma 6. For a ∈ R^{d}, v(x, a) is differentiable in x with derivative

v′(x, a) = E[U′(x + a^{⊤} · R, I_{1})]. The proof is obvious since Ω is finite.

Theorem 7. V (x) is differentiable in x and V′(x) = v′(x, a∗) where a* = a ∗ (x) is the maximum point in L of the function R^{d} a ↦ v(x, a) . Proof. From Lemmata 4 and 5 we know that a unique maximum point a* exists in L . Now V (x±)−V (x) ≥ v(x± , a∗)−v(x, a∗). Hence v′_{+}(x, a∗) ≤ V′_{+}(x) ≤ V′_{−}(x) ≤ v′_{−}(x, a∗) where V′_{±}(x) and v′_{±}(x, a∗) denote the right and left derivatives, respectively. Now Lemma 6 applies. [] Lemma 8. For x > 0, v(x, a) is partially differentiable in a ∈ R^{d} with partial derivatives

∂_{k}v(x, a) = E[U′(x + a^{⊤} · R, I_{1}) · R^{k}] . Again, the proof is obvious since Ω is finite.

Theorem 9. Let be x ∈ R, then E[U′(x + a ∗^{⊤} ·R, I_{1}) · R^{k}] = 0 for 1 ≤ k ≤ d where a* is the unique maximum point in L of the function R^{d} a ↦ v(x, a) . Proof. The function a ↦ v(x, a) is partially differentiable by Lemma 8. Therefore we know for the maximum point a* that ∂_{k}v(x, a∗) = 0 for 1 ≤ k ≤ d. []

Corollary 10. If U′(x, i) > 0 for all x and i or more generally if U′(x + a ∗^{⊤} ·R, I_{1}) > 0 on Ω, then

the probability measure Q defined by dQ = const·U′(x+a∗^{⊤}·R, I_{1})dP is an equivalent martingale measure.

5 The multi-period model We remind the reader that R_{n} = ρ_{n}(I_{n−1}, I_{n}) for some function ρ_{n}. Now we need the support of R_{n} given I_{n−1} = i defined by ∑_{n}(i) := {ρ_{n}(i, j); j ∈ E_{n}, p_{ij}(n) > 0}, i ∈ E_{n−1}, n ≥ 1. L_{n}(i) is the smallest linear space in R^{d} containing ∑_{n}(i), i.e. L_{n}(i) is the smallest linear space L in R^{d} such that P [ R_{n} ∈ L|I_{n−1} = i] = 1. Then for ∑ and L as defined in §4 we have ∑ = ∑_{1}(i_{0}) and L = L_{1}(i_{0}). We may assume (w.l.o.g.) that the no-arbitrage condition also holds locally (see Dalang et al. [3] Lemma 2.3 , Pliska [31](3.22), Schäl [37]§2) , i.e., (NA)∗ “a^{⊤} · σ ≥ 0∀σ ∈ ∑_{n}(i)” implies “a^{⊤} · σ = 0∀σ ∈ ∑_{n}(i)” , i ∈ E_{n−1} , a ∈ R^{d}. The condition (NA)* just means:

“P [ a^{⊤} · R_{n} ≥ 0|I_{n−1} = i] = 1” implies “P [ a^{⊤} · R_{n} = 0|I_{n−1} = i] = 1” , i ∈ E_{n−1} , a ∈ R^{d}. As in §4, we will use the assumption U ∈ F_{N} for the utility function U. We recall that by (2.8): X_{N}^{ϕ}(x) = X_{n}^{ϕ}(x) + ∑_{m=n+1}^{N} ϕ_{m−1}^{⊤} · R_{m} and we now define: (5.1) v_{n}(x, i, ϕ) = E[ U(x + ∑_{m=n+1}^{N} ϕ_{m−1}^{⊤} · R_{m} , I_{N})|I_{n} = i ],

V_{n}(x, i) = sup_{ϕ} v_{n}(x, i, ϕ) ,

v_{N}(x, i, ϕ) = V_{N}(x, i) = U(x, i) for all ϕ. There v_{n}(x, i, ϕ)[V_{n}(x, i)] is the [maximal] expected utility of the terminal wealth given the market information i and the discounted wealth x at time n. Since I_{0} = i_{0} is fixed, we can set: (5.2) v_{0}(x, ϕ) = v_{0}(x, i_{0}, ϕ), V_{0}(x) = V_{0}(x, i_{0}). Obviously we have an N-stage Markov decision model with no running costs and terminal reward U. Let us now introduce the well-known reward operator: (5.3) T_{n}^{a}f(x, i) := E[f(x + a^{⊤} · R_{n+1} , I_{n+1})|I_{n} = i] for any f : R × E_{n+1} ↦ R , a ∈ R^{d}, i ∈ E_{n}. We can express T_{n}^{a}f(x, i) in (5.3) by the transition matrix (p_{ij}(n)) according to (5.4) T_{n}^{a}f(x, i) = ∑_{j∈E_{n+1}} p_{ij}(n) · f(x + a^{⊤} · ρ_{n+1}(i, j), j) Thus for fixed i, T_{n}^{a}f(x, i) can be expressed by an ordinary expectation and we can use the results of §4. The following equation is sometimes called fundamental equation (Dynkin & Yushkevich [6]) and follows from Fubini’s theorem (Hinderer [18] Lemma 11.1). (5.5) v_{n}(x, i, ϕ) = T_{n}^{ϕ_{n}(x,i)}v_{n+1}(x, i, ϕ) , n ≥ 0 .

Now we can give the well-known optimality equation (see Hinderer 1970, Theorems 14.4, 18.4): (5.6) V_{n}(x, i) = sup_{a∈R^{d}} T_{n}^{a}V_{n+1}(x, i) = sup_{a∈L_{n+1}(i)} T_{n}^{a}V_{n+1}(x, i) , n ≥ 0.

Proposition 11. Let be f ∈ F_{n+1} and 0 ≤ n < N. Then:

(a) (x, a) ↦ T_{n}^{a}f(x, i) is continuous;

(b) there exists a unique measurable function δ : R × E_{n} ↦ R^{d} such that δ(x, i) ∈ L_{n+1}(i) and T_{n}^{δ(x,i)}f(x, i) = max_{a∈R} T_{n}^{a}f(x, i) =: F (x, i);

(c) F (x, i) is strictly concave in x. Proof. Part (a) and part (b) follow from Lemma 4. The remaining properties follow from Lemma 5. [] Finally, we will use another assumption which will be discussed in §6 for special cases. Assumption 12. For V′_{n}(x, i) = ∂V_{n}(x, i)/∂x we have: V′_{n}(−∞, i) > 0 and V′_{n}(+∞, i) ≤ 0 , 0 ≤ n < N. The existence of V′_{n}(x, i) follows from: Lemma 13. (a) V_{n}(x, i) is differentiable in x with derivative V′_{n}(x, i).

(b) V_{n}(x, i) ∈ F_{n} for 0 ≤ n ≤ N. Proof. Part (a) can be proved as Theorem 7. By assumption we know that V_{T} = U is in F_{N}. Upon using the optimality equation (5.6), this statement follows by backward induction from Assumption 12 and Proposition 11c. [] Now we obtain from (5.5), (5.6), Proposition 11, and Lemma 13 by the usual arguments of dynamic programing (Feinberg [9]) : Theorem 14. There exists a unique policy ϕ∗ = (ϕ∗ n) such that for 0 ≤ n < N :

tcol

ϕ∗ n(x, i) ∈ L_{n+1}(i) and tcol

T_{n}^{ϕ∗ n(x,i)}V_{n+1}(x, i) = max_{a∈R^{d}} T_{n}^{a}V_{n+1}(x, i) = V_{n}(x, i), v_{n}(x, i, ϕ∗) = V_{n}(x, i).

Proof. We just have to define ϕ_{n} = δ as in Proposition 11b where f = V_{n+1}. []

Proposition 15. For some x > 0, let ϕ∗ be the optimal policy of Theorem 14. Then

V′_{0}(x) = E[U′(X_{N}^{ϕ∗}(x), I_{N})]. Proof. By induction we are going to prove (5.7) V′_{0}(x) = E[V′_{n}(X_{n}^{ϕ∗}(x), I_{n})] , 1 ≤ n ≤ N. For n = 1 we have V_{0}(x) = max_{a∈L_{1}(i_{0})} E[V_{1}(x + a^{⊤} · R_{1}, I_{1})] where ϕ_{0}(x, i_{0}) is a maximum point. In view of Lemma 13, V_{1}(x, i) satisfies the Assumption 3 for U(x, i) (for N = 1). With the help of Lemma 6 and Theorem 7, we can conclude:

V′_{0}(x) = v′_{0}(x, ϕ∗ 0(x, i_{0})) = E[V′_{1}(x + ϕ∗ 0(x, i_{0})^{⊤} · R_{1}, I_{1})] tcol tcol

= E[V′_{1}(X_{1}^{ϕ∗}(x), I_{1})] . Hence (5.7) holds for n = 1. Now assume that (5.7) holds for n − 1. For fixed i ∈ E_{n−1} we have

V_{n−1}(x, i) = max_{a∈L_{n}(i)} E[V_{n}(x + a^{⊤} · R_{n}, I_{n})|I_{n−1} = i]

where ϕ∗ n_{−1}(x, i) is a maximum point. In view of Lemma 13, V_{n}(x, i) satisfies the Assump-

tcol

tion 3 for U(x, i) (for N = n). As above we obtain:

V′_{n−1}(x, i) = E[V′_{n}(x + ϕ∗ n_{−1}(x, i)^{⊤} · R_{n}, I_{n})|I_{n−1} = i]. tcol

Then by (2.8) : X_{n}^{ϕ∗}(x) = X_{n−1}^{ϕ∗}(x) + ϕ∗ n_{−1}^{⊤} · R_{n} tcol

Since X_{n−1}^{ϕ∗}(x) is a function of (I_{1}, ...I_{n−1}), it follows that (5.8) V′_{n−1}(X_{n−1}^{ϕ∗}(x), I_{n−1})

= E[V′_{n}(X_{n−1}^{ϕ∗} + ϕ∗ n_{−1}(x, i)^{⊤} · R_{n}, I_{n})|σ(I_{1}, ..., I_{n−1})] tcol

= E[V′_{n}(X_{n}^{ϕ∗}, I_{n})|σ(I_{1}, ..., I_{n−1})] . By assumption we get V′_{0}(x) = E[V′_{n−1}(X_{n−1}^{ϕ∗}(x), I_{n−1})] = E[E[V′_{n}(X_{n}^{ϕ∗}, I_{n})|σ(I_{1}, ..., I_{n−1})]] = E[V′_{n}(X_{n}^{ϕ∗}(x)), I_{n})] and (5.7) follows. Now we can use (5.7) for n = N and finally obtain the result. []

Theorem 16. Assume that U′ is positive or more generally that U′(X_{N}^{ϕ∗}(x), I_{N}) is positive where ϕ∗ is the optimal policy of Theorem 14. Define for some x the process {Z_{n}, 0 ≤ n ≤ N} by

Z_{n} := V′_{n}(X_{n}^{ϕ∗}(x), I_{n}) , in particular Z_{0} = V′_{0}(x), Z_{N} = U′(X_{N}^{ϕ∗}(x), I_{N}), then:

(a) One obtains an equivalent martingale measure Q_{x}^{U} on Ω by

Q_{x}^{U}[{ω}] = − V− ; ′ ; 01 ; − ; (− x− )Z_{N}(ω)P [{ω}] = (Z_{N}(ω)/Z_{0})P [{ω}].

tcol tcol tcol tcol

tcol

(b) {Z_{n}/Z_{0}, 0 ≤ n ≤ N} is a martingale under P and is called the density process of dQ_{x}^{U}/dP .

(c) If we write E_{x}^{U}[...] for the expectation under Q_{x}^{U} , then for any function f_{n} E_{x}^{U}[f_{n}(I_{1}, ..., I_{n})|σ(I_{1}, ..., I_{n−1})] = E[f_{n}(I_{1}, ..., I_{n}) · Z_{n}|σ(I_{1}, ..., I_{n−1})] / Z_{n−1}.

Proof. If U′(X_{N}^{ϕ∗}(x), I_{N}) is positive then we know from Proposition 15 that Q_{x}^{U} is indeed a probability measure which is equivalent to P. Now part (b) immediately follows from (5.8) and part (c) is a consequence of Bayes’ rule. From Theorem 9 we conclude that

(5.9) E[V′_{n}(x + ϕ∗ n_{−1}(x, i)^{⊤} · R_{n}, I_{n}) · R_{n}^{k}|I_{n−1} = i] = 0 for 1 ≤ k ≤ d tcol

since ϕ∗ n_{−1}(x, i) is the unique maximum point in L_{n}(i) of the function

tcol

R^{d} a ↦ E[V′_{n}(x + a^{⊤} · R_{n}, I_{n})|I_{n−1} = i] . ¿From (5.9) we obviously obtain as in proof of (5.8):

E[R_{n}^{k} · Z_{n}|σ(I_{1}, ..., I_{n−1})] = 0 , 1 ≤ k ≤ d.

Finally from (c) we get (5.10) E_{x}^{U}[R_{n}^{k}|σ(I_{1}, ..., I_{n−1})] = 0 , 1 ≤ n ≤ N, 1 ≤ k ≤ d.[]

6 Applications In this section we will show that all assumptions are satisfied for the case where (6.1) U(x, i) = −e^{−γx} · L(i) for some γ > 0 and some function L : E_{N} ↦ (0, ∞).

A classical example is U(x, i) = −1 ; − ; γe^{−γx} (or U(x, i) = 1 ; − ; γ(1 − e^{−γx})). If one wants to

tcol

tcol

consider the utility of the terminal wealth itself rather that than the discounted terminal wealth, then one can choose U_{B}(x, i) := U(B_{N} · x, i) which has the same form (6.1). A further interesting example was studied by Grandits & Rheinländer [15]. They look for a policy ϕ^{0} such that (6.2) E[exp{γ · (X̆ − X_{N}^{ϕ^{0}}(x))}] = inf_{ϕ} . Instead of super-hedging the contingent claim X one tries to choose ϕ^{0} for the given initial wealth x such that the expected loss is minimized. There the positive values of X̆ −X_{N}^{ϕ^{0}}(x) are given more weight than the negative values. Of course, this is the view of the seller. If one chooses I_{N} = S_{N} and X̆ = f(S_{N}) as in the case of a European call option, then this

problem is included in the framework of (6.1) by choosing L(i) := 1 ; − ; γ exp{γ · f(i)}. There

tcol

are some recent papers treating the problem E[ℓ({X̆ −X_{N}^{ϕ^{0}}(x)}^{+})] = inf_{ϕ} (see Runggaldier & Zaccaria [34] for further references and an approach via dynamic programming in the case of transaction costs). For convenience, we only will consider the case γ = 1. Lemma 17. In the situation (6.1) with γ = 1, one has:

(a) Assumption 3 is satisfied and U′(x, i) = e^{−x} · L(i) is positive;

(b) V_{n}(x, i) = −e^{−x} · L_{n}(i) for some function L_{n} : E_{n} ↦ (0, ∞);

(c) for the optimal policy ϕ∗ of Theorem 14, ϕ∗ n(x, i) does not depend on x;

tcol

(d) Assumption 12 is satisfied. Proof. Part (a) is obvious and part (d) follows from part (b). Part (b) is obvious for n = N. Now assume that (b) holds for n + 1. Then according to Theorem 14 V_{n}(x, i) = max_{a∈L_{n+1}(i)} − ∑_{j∈E_{n+1}} p_{ij}(n) exp{−x−a^{⊤}·ρ_{n+1}(i, j)}·L_{n+1}(j) =: −e^{−x}·L_{n}(i) where the maximum point ϕ_{n}(i) is indeed independent of x and (6.3) L_{n}(i) = min_{a∈L_{n+1}(i)} ∑_{j∈E_{n+1}} p_{ij}(n) exp{−a^{⊤} · ρ_{n+1}(i, j)} · L_{n+1}(j). [] Consider the case where R_{1}, ..., R_{N} are independent and I_{n} = R_{n}. Then both p_{ij}(n) and ρ_{n+1}(i, j) are independent of i. From the proof of Lemma 17 we conclude that in that case ϕ∗ n(x, i) =: ϕ∗ n neither depends on x nor on i.

tcol

tcol

Definition 18. In the situation (6.1) with γ = 1, let ϕ∗ be the optimal policy of Theorem 14 which is independent of the initial wealth x according to Lemma 17. Then X∗ n := X_{n}^{ϕ∗}(0)

tcol

, n = 0, ..., N is called the optimal wealth process. Lemma 19. In the situation (6.1) with γ = 1, the density process of Theorem 16 is given by

Z_{n}/Z_{0} = exp{−X∗ n} · L_{n}(I_{n})/L_{0}(i_{0}), 0 ≤ n ≤ N, where Z_{N}/Z_{0} = exp{−X∗ N} · tcol tcol

L(I_{N})/L_{0}(i_{0}). Proof. We have Z_{n} = V′_{n}(x + X∗ n , I_{n}) = e^{−x} · exp{−X∗ n} · L_{n}(I_{n}) , thus Z_{0} = e^{−x} · L_{0}(i_{0})

tcol

tcol

and Z_{N} = e^{−x} · exp{−X∗ N} · L(I_{N}). [] tcol

Now we obtain from Theorem 16 the following result: Theorem 20. Let {X∗ n} be the optimal wealth process in the situation (6.1) with γ = 1

tcol

and let L_{n} be defined by (6.3) where L_{N} = L. Then Q^{U} defined by

Q^{U}[{ω}] := exp{−X∗ N(ω)} · L(I_{N}(ω))/L_{0}(i_{0})P [{ω}] is a martingale measure.

tcol

Upon choosing L(i) = 1 , for example, we obtain the existence of a special martingale measure and the Fundamental Theorem of §3 is proved where (NA) was used in the proof of Lemma 4c. Then the resulting martingale measure Q^{E} has the form:

(6.4) Q^{E}[{ω}] = const · exp{−X∗ N(ω)}P [{ω}] . tcol

There Q^{E} is the unique solution of the minimum problem where the relative entropy I(Q, P) of Q w.r.t. P has to be minimized (Fritelli [13], Grandits [14]). Q^{E} may also be considered as a multi-stage Esscher transform of P whereas in Bühlmann et al. [2] the Esscher transform is used locally for each time and each history in order to construct a martingale measure.

Another important example is the following: (6.5) U(x, i) = L^{(0)}(i) + 2 L^{(1)}(i) · x − L^{(2)}(i) · x^{2}

for some function L^{(k)} : E_{N} ↦ R , k = 0, 1, 2 where L^{(2)} is positive. An interesting special case is the best hedging policy for hedging the contingent claim X. One looks for a policy ϕ^{o} such that (6.6) E[{X̆ − X_{N}^{ϕ^{o}}(x)}^{2}] = inf_{ϕ} . Thus, instead of super-hedging X one tries to choose ϕ^{o} for the given initial wealth x such that the expected quadratic loss is minimized. Now, in contrast to (6.2), the positive values of X̆ −X_{N}^{ϕ^{o}}(x) have the same weight as the negative values. This is fair both from the point of view of the seller and the buyer. If one again chooses I_{N} = S_{N} and X̆ = f(S_{N}), then this problem is included in the framework of (6.5) by choosing L^{(0)}(i) = f(i)^{2}, L^{(1)}(i) = f(i), and L^{(2)}(i) := 1. Now, one can look for that initial wealth x̂ such (6.7) inf_{ϕ} E[{X̆ − X_{N}^{ϕ}(x̂)}^{2}] = min_{x} inf_{ϕ} E[{X̆ − X_{N}^{ϕ}(x)}^{2}] . The optimal value x̂ is called fair hedging price in [36] and approximation price by Schweizer [40] for the contingent claim X. Whereas [36] relies on stochastic dynamic programming, Schweizer [40] uses martingale methods. Lemma 21. In the situation (6.5) one has:

(a) Assumption 3 is satisfied ;

(b) V_{n}(x, i) = L_{n}^{(0)}(i) + 2 L_{n}^{(1)}(i) · x − L_{n}^{(2)}(i) · x^{2} for some function L_{n}^{(k)} : E_{n} ↦ R , k = 0, 1, 2, where L_{n}^{(2)} is positive;

(c) Assumption 12 is satisfied. Proof. Part (a) is obvious and part (c) follows from part (b). Part (b) is obvious for n = N. Now assume that (b) holds for n + 1. Then according to Theorem 14 V_{n}(x, i) = max_{a∈R^{d}} ∑_{j∈E_{n+1}} p_{ij}(n + 1)[L_{n+1}^{(0)}(j) + 2 L_{n+1}^{(1)}(j) · {x + a^{⊤} · ρ_{n+1}(i, j)}

−L_{n+1}^{(2)}(j){x + a^{⊤} · ρ_{n+1}(i, j)}^{2}]
= L̃^{(0)}(i) + 2L̃^{(1)}(i)x − L̃^{(2)}(i)x^{2} + L ∗ (x, i) − L̂(x, i)

where L̃^{(k)}(i) := ∑_{j} p_{ij}(n + 1)L_{n+1}^{(k)}(j) , k = 0, 1, 2,

L∗(x, i) := − min_{a∈R^{d}} ∑_{j} p_{ij}(n+1)·L_{n+1}^{(2)}(j){L_{n+1}^{(1)}(j)/L_{n+1}^{(2)}(j)−x−a^{⊤} ·ρ_{n+1}(i, j)}^{2},

L̂(x, i) := ∑_{j} p_{ij}(n + 1)L_{n+1}^{(2)}(j){L_{n+1}^{(1)}(j)/L_{n+1}^{(2)}(j) − x}^{2}.

Now L̂ has the desired form. This is also true for L∗ as follows from [36] Proposition 7.3 for the case d = 1 and is easily extended to the case d ≥ 1. [] The problem with this example (6.5) is that U′(x, i) = 2 L^{(1)}(i)−2 L^{(2)}(i)·x is not positive everywhere and also U′(X_{N}^{ϕ∗}(x), I_{N}) need not to be positive everywhere. Therefore the measure Q_{x}^{U} is in general only a signed measure and is called signed martingale measure because it has all other properties of a martingale measure. In Schweizer [40], [41] the relations to the mean-variance frontier, the variance-optimal and the minimal martingale measures are explained. Schweizer [41] also considers the general d-dimensional case in a general semi-martingale framework which includes the discrete-time case as special case. Motoczynski [28] studies the discrete-time d-dimensional setting. Grandits [14] constructs the p-optimal martingale measure as a generalization of the variance-optimal martingale measure by replacing L^{2} by L^{p} for some p > 1. For the construction Grandits also uses methods of dynamic programming similar to those explained in § 5. Example. The multi-dimensional Binomial model. For convenience we assume T = 1 and write R^{k} := R_{1}^{k}, 1 ≤ k ≤ d. We consider the model where

R^{k} takes on values in {−α^{k}, β^{k}} where α^{k}, β^{k} > 0.

We assume w.l.o.g. that I_{1} = R_{1} = R. For d = 1 the model is complete and is also called Cox-Ross- Rubinstein model. The unique martingale measure P ∗ is given by P ∗ [R^{1} = i] = q∗ 1(i) where

tcol

q∗ k(−α^{k}) := β^{k}/(α^{k} + β^{k}) , q∗ k(β^{k}) := α^{k}/(α^{k} + β^{k}). tcol tcol

For d > 1, the model is no longer complete since R takes on 2^{d} values where 2^{d} > 1 + d (Jacod & Shiryaev [20]). It is easy to see that Q ∗ [R = (i^{1}, ..., i^{d})] = Π_{k=1}^{d}q∗ k(i^{k}) defines a

tcol

martingale measure for d ≥ 1 which seems to be a natural one if the components R^{k} of R are independent under P. Infact in that case this measure coincides with Q^{E} as defined in (6.4) but is different from the so-called minimal martingale measure which is obtained by the (discrete-time) Girsanov transformation and which coincides with the variance-optimal martingale measure for T = 1 (Schweizer [40]). Q∗ also differs from the martingale measure obtained by the so-called numéraire portfolio (Korn & Schäl [23]). We want to prove:

Q∗ = Q^{E} if the components R^{k} of R are independent under P. We first observe that P ∗ = Q∗ = Q^{E} if d = 1 since then there is only one martingale measure. Hence for inf_{a∈R} E[exp{a · R^{k}}] = E[exp{a∗ k · R^{k}}] =: A_{k} we know that q∗ k(i) =

tcol

tcol

exp{a∗ k · i}/ A_{k}. Now inf_{a∈R^{d}} E[exp{x + a^{⊤}R}] = e^{x} · inf_{a∈R^{d}} Π_{k=1}^{d}E[exp{a^{k} · R^{k}}] tcol

= e^{x} · Π_{k=1}^{d} inf_{a∈R} E[exp{a · R^{k}}] = e^{x} · Π_{k=1}^{d}A_{k} by independence. Therefore we obtain

Q^{E}[{(i^{1}, ..., i^{d})}] / P [{(i^{1}, ..., i^{d})}] = exp{x + ∑_{k=1}^{d} a∗ k · i^{k}}/(e^{x} · Π_{k=1}^{d}A_{k}) tcol

= Π_{k=1}^{d} exp{a∗ k · i^{k}}/ A_{k} = Π_{k=1}^{d}q∗ k(i^{k}). tcol tcol

The result immediately extends to T > 1 if one assumes that R_{1}^{1}, ..., R_{1}^{d}, ..., R_{T}^{1} , ..., R_{T}^{d} are independent. Motoczynski & Stettner [29] consider super-hedging (see next sections) in the multi- dimensional multi-period Cox-Ross-Rubinstein model. [] 7 Super-hedging, the one-period model. In this section we restrict attention to the case N = 1 and study values x for the initial wealth that allow for super-hedging a given discounted contingent claims X̆ by some policy ϕ such that X_{N}^{ϕ}(x) ≥ X̆. As in §4 we set R := R_{1}. Then we have to solve the problem: (7.1) x∗ = inf{x ∈ R; ∃a ∈ R^{d}s.th. x + a^{⊤} · R(ω) ≥ X̆(ω) , ω ∈ Ω} . Since ω ∈ Ω is finite, one obtains a linear program by introducing: (7.2) V := {(x, a) ∈ R^{1+d}; 1 · x + ∑_{k} a^{k} · R^{k}(ω) ≥ X̆(ω) , ω ∈ Ω}. Then (7.1) can be written as the linear program: (P) min{x; (x, a) ∈ V } . Then the dual program is: (D) max{∑_{ω∈Ω} q(ω) · X̆(ω) ; q ∈ Q_{1}} where (7.3) Q̂_{1} := {q ∈ R^{|Ω|}; q(ω) ≥ 0, ω ∈ Ω, ∑_{ω} q(ω) = 1, ∑_{ω} q(ω) · R^{k}(ω) = 0, 1 ≤ k ≤ d}. Q̂_{1} can be identified with set of martingale measures in the case N = 1. A useful characterization is given in Pliska [31] p. 59. The set of equivalent martingale measures is (7.4) Q_{1} := {q ∈ Q̂ ; q(ω) > 0, ω ∈ Ω}. Obviously, V is not empty; one only has to choose x large enough. From the Fundamental Theorem in §3 we know that Q_{1} and hence Q̂_{1} is not empty under the no-arbitrage condition. Now the duality theorem of linear programming applies and we obtain (Pliska [31] p. 27):

Proposition 21. x∗ = min{x ∈ R; ∃a ∈ R^{d}s.th. x + a^{⊤} · R(ω) ≥ X̆(ω) , ω ∈ Ω}

= max_{q∈} Q̂_{1} ∑_{ω∈Ω} q(ω) · X̆(ω) = sup_{q∈Q_{1}} ∑_{ω∈Ω} q(ω) · X̆(ω) =: sup_{q∈Q_{1}} E_{q}[X̆]. tcol

The latter equality holds because Q_{1} is dense in Q̂_{1}. This is theorem 2 in the case N = 1. An immediate consequence is:

Corollary 22. E_{q}[X̆] ≤ 0∀q ∈ Q_{1} if and only if there exists some a ∈ R^{d} such that a^{⊤} · R ≥ X̆ . 8 Super-hedging, the multi-period model. In this section we consider the general case N ≥ 1 and we may assume (NA)* as in §5. It will be convenient to work with a canonical probability space: (8.1) Ω := Ω_{N} where Ω_{n} := E_{0} × ... × E_{n}, ,

I_{n}(ω) = i_{n} , ω_{n} := (i_{0}, ..., i_{n}) for ω = (i_{0}, ..., i_{N}) ∈ Ω. We need the following characterization of Q([33], Korn & Schäl [23]). Proposition 23. Q ∈ Q if and only if (8.2) Q[{ω}] = q_{0}(i_{0}; i_{1}) · q_{1}(i_{0}, i_{1}; i_{2}) · ... · q_{N−1}(ω_{N−1}; i_{N})

where q_{n}(ω_{n}; ·) ∈ Q_{n}(i_{n}) for ω = (i_{0}, ..., i_{N}) and (8.3) Q_{n}(i) := {q; q : E_{n+1} ↦ (0, 1); ∑_{j∈E_{n+1}} q(j) = 1 , ∑_{j∈E_{n+1}} q(j) · ρ(i, j) = 0}, i ∈ E_{n}. There is now an interesting relation to dynamic programming. One can look upon Q_{n}(i) as a set of actions available at time n given the information i about the market at time n. Moreover, a function q_{n} can be considered as a function which selects for each given history ω_{n} an action q_{n}(ω_{n}; ·) ∈ Q_{n}(i_{n}). Then a policy is given by (q_{0}, q_{1}, ..., q_{N−1}) and defines a martingale measure through (8.2). In models with a more general space Ω, the situation is more complicated. But even then the set Q enjoys a property ([37] Lemma 1.8) which is known for the set of policies in dynamic programming and is called “to admit needle-like variation” by Fakeev [8], “stability” by Hinderer [18], and “product property” by Hordijk ]19]. This relation to dynamic programming was used by El Karoui & Quenez [7] for diffusion models and by Naik & Uppal [30]. As mentioned above, it is natural from the point of view of Markov decision theory to generalize the concept of an American option. We call the generalization a dynamic option which has interesting applications. The buyer has to pay a premium x to the seller and then he can choose any policy δ according to a certain dynamic program. Then the policy δ implies some (non- discounted) cost or claim X^{δ} := B_{N} · X̆^{δ} : Ω ↦ R which the seller has to pay to the buyer at N. On the other side, the seller can invest according to a policy ϕ in the market, i.e. the seller is the investor. Now, we look for some initial wealth x such that (8.4) for each policy δ of the buyer there is some policy ϕ(δ) of the seller such that X_{N}^{ϕ(δ)}(x) ≥ X̆^{δ} on Ω, i.e. such that X̆^{δ} can be super-hedged by x and ϕ(δ). In particular, we are again interested in the smallest value x∗ such that (8.4) holds for x = x ∗ . We obtain the usual European option if there exists only one policy for the buyer. One obtains an American call option if the buyer may choose a stopping time τ : Ω ↦ {0, ..., N} and may exercise his option at time τ . Such a contract is equivalent to a payment X^{τ} = (S_{τ}^{1} − χ)^{+}. Then one is interested in finding some initial wealth x and some policy ϕ such that for the discounted claim: (8.5) X_{τ}^{ϕ}(x) ≥ X̆^{τ} = (S̆_{τ}^{1} − χ/B_{τ})^{+} = (S_{τ}^{1} − χ)^{+}/B_{τ} for all stopping times τ. We can write X_{τ}^{ϕ}(x) = X_{N}^{ϕ(τ)}(x) where ϕ_{n}(τ) := ϕ_{n} for n ≤ τ and ϕ_{n}(τ) := 0 for n > τ. There, first the buyer decides to stop and then the seller decides to stop investing. Thus, the American option fits into the framework of a dynamic option and the investment policy of the seller indeed depends on τ which is not clear from (8.5) at first glance. A necessary condition for (8.4) can be deduced from (3.3) (i.e. E_{Q}[X_{N}^{ϕ}(x)] = x for any Q ∈ Q). We obtain: (8.6) x̄ := sup_{Q,δ} E_{Q}[X̆^{δ}] ≤ x∗ := inf{x; (8.4) holds }. Now our goal is to show that (8.7) ∀δ∃ϕ(δ) such that X_{N}^{ϕ(δ)}(x̄) ≥ X̆^{δ}. Then x̄ = x∗ and the infimum in the definition of x∗ is attained. First we describe a dynamic program for the buyer by the usual set up. Here it is useful to choose a model with no running costs and a terminal reward (claim) X̆ which depends on the whole history at time N. [In such a framework one can model restrictions on the admissible actions by the choice of the terminal reward.] In order to describe general stopping times of the buyer by policies of the buyer we have to allow for non-Markovian policies.

(1) A =′_{n} is the action space of the buyer;

tcol

(2) H_{n} := E_{0} × A′_{0} × ... × E_{n} is the space of histories of the buyer up to epoch n;

(3) a policy δ = (δ_{0}, ..., δ_{N−1}) of the buyer is given by functions δ_{n} : Ω_{n} ↦ A =′_{n} ;

tcol

(4) X̆ : H_{N} ↦ R is a discounted claim function where we write X̆^{δ}(ω) := X̆(h_{N}^{δ}(ω)) and h_{n}^{δ}(ω_{n}) := (i_{0}, δ_{0}(i_{0}), i_{1}, ..., δ_{n−1}(ω_{n−1}), i_{n}) ∈ H_{n}.

At each time n the buyer can choose an action a′_{n} according to some policy δ and the seller will immediately be informed about the choice of a′_{n} at time n. The claim of the dynamic option is executed at time N; at the end the seller has to pay the amount B_{N} · X̆^{δ}(ω) to the buyer. For an American call option, one will have A =′_{n} := {0, 1} where ’1’ means ’to stop’ and

tcol

’0’ stands for ’to do nothing’. Moreover, one can choose X̆(i_{0}, a′_{0}, i_{1}, ..., a′_{N−1}, i_{N}) = (i_{τ} − χ)^{+}/B_{τ} where τ := inf{n; a_{n} = 1} with inf ∅ = N. One can imagine that the claim (i_{τ} − χ)^{+} at time τ is invested in the savings account up to time N; at N the amount (i_{τ} − χ)^{+} · B_{N}/B_{τ} is paid to the buyer including interest for the time between τ and N. Discounting then leads to (i_{τ} − χ)^{+}/B_{τ}. Thus the American option can also be described by a claim executed at time N. We will present another example.

Example from the bond market. Assume that at time n = 0 the buyer gives some capital y_{0} to the seller. In that situation, an action a’ consists in taking back the amount a’. Then

A =′_{n} = [0, ā_{n}] for some ā_{n} ≥ 0 and for n ≥ 0.

tcol

Actually admissible actions a′_{n} at time n are only those with a′_{n} ≤ α_{n} where

α_{n} := ā_{n} ∧ (y_{0} − a′_{1} − ... − a′_{n−1}) , n < N. This can be modeled by either defining X̆(i_{0}, a′_{0}, i_{1}, ..., a′_{N−1}, i_{N}) = −∞ if some a′_{n} is not admissible or by setting X̆(i_{0}, a′_{0}, i_{1}, ..., a′_{N−1}, i_{N}) = X̆(i_{0}, a′_{0} ∧ α_{0}, i_{1}, ..., a′_{N−1} ∧ α_{N−1}, i_{N}). In the later case the actions a′_{n} and a′_{n} ∧ α_{n} are identified. [] A further modern example is provided by a passport option (see Delbaen & Yor [5], Shreve & Vecer [42] which allows the buyer to take (long or short) positions in a stock. If at N the buyer makes a benefit, he can keep it. Otherwise he does not have to pay for the losses. The idea of our approach consists in writing x̄ := sup_{Q,δ} E_{Q}[X̆^{δ}] as value of a dynamic program in a super-model with a super-player combining the buyer choosing δ (controlling the claim) and the market choosing the martingale measure Q (controlling the law of motion). As before, the initial state i_{0} is fixed. The data are

(ˆ1) ; Â ; =_{n} := A =′_{n} × {q : E_{n+1} ↦ (0, 1), ∑_{i∈E_{n+1}} q(i) = 1}, n ≥ 0, tcol tcol

(ˆ2) ; Â ; =_{n}(i_{n}) = ; Â ; =′_{n} × Q_{n}(i_{n}) for n ≥ 0,

tcol

tcol

(ˆ3) δ̂:= (δ, Q) where δ = (δ_{0}, ..., δ_{N−1}), Q = (q_{0}, ..., q_{N−1}), (ˆ4) X̂ := X̆, X̂^{δ} := X̆^{δ}; (ˆ5) the law of motion is given through Q as in Proposition 23. A policy δ is called admissible at h_{n} = (i_{0}, a′_{0}, ..., i_{n}) if δ_{m}(i_{0}, ...i_{m}) = a′_{m} for m < n. Then the value functions of the super-model are given by (8.8) V̂_{n}(h_{n}) := sup{E_{Q}[X̆^{δ}|I_{0} = i_{0}, ..., I_{n} = i_{n}]; δ is admissible at h_{n}, Q ∈ Q},

for h_{n} = (i_{0}, a′_{0}, ..., i_{n}) ∈ H_{n}. Then V̂_{0}(i_{0}) := x̄. An important tool will be the optimality equation (Hinderer [18], Theorem 14.4): (8.9) V̂_{n}(h_{n}) = sup_{a′∈A =′_{n},q∈Q_{n}(i_{n})} ∑_{i∈E_{n+1}} q(i)V̂_{n+1}(h_{n}, a′, i),

which implies:

Lemma 24. ∑_{i∈E_{n+1}} q(i)[V̂_{n+1}(h_{n}, a′, i) − V̂_{n}(h_{n})] ≤ 0∀a′ ∈ A =′_{n}, q ∈ Q_{n}(ω_{n}). tcol

Now we can use the results for the one-period model by use of the standard reduction method of Hinderer [18] p. 24. The idea consists in treating time n as new origin for fixed (h_{n}, a′). Then Q_{n}(i_{n}) is the set of equivalent martingale measures for the new one-period model. Now Corollary 22 applies and we obtain:

Proposition 25. For fixed (h_{n}, a′) = (i_{0}, a′_{0}, ..., i_{n}, a′) there exists some ϕ̄_{n}(h_{n}, a′) such that

V̂_{n+1}(h_{n}, a′, i) − V̂_{n}(h_{n}) ≤ ϕ̄_{n}(h_{n}, a′)^{⊤} · ρ_{n+1}(i_{n}, i)∀i ∈ E_{n+1}.

One can interpret ϕ̄_{n}(h_{n}, a′) as the decision of the seller at time n about the investment in the market after the buyer chose action a’. Therefore the super-model can be looked upon as a stochastic game with complete information in the sense of Küenle [25]. Now we can construct the desired investment policy, which will be non-Markovian, as follows: (8.10) ϕ̄_{n}^{δ}(ω_{n}) := ϕ̄_{n}(h_{n}^{δ}(ω_{n}), δ_{n}(ω_{n})), n ≥ 0, where h_{0}^{δ}(ω_{0}) = ω_{0} = i_{0}. There h_{n}^{δ}(ω_{n}) is defined above. Then one has by Proposition 24: (8.11) V̂_{n+1}(h_{n}^{δ}(ω_{n}), δ_{n}(ω_{n}), i_{n+1}) − V̂_{n}(h_{n}^{δ}(ω_{n})) ≤ ϕ̄_{n}^{δ}(ω_{n})^{⊤} · ρ_{n+1}(i_{n}, i_{n+1}). Now we can prove our goal (8.7): Theorem 26. X_{N}^{ϕ̄(δ)}(x̄) ≥ X̆^{δ} where ϕ̄(δ) := (ϕ̄_{n}^{δ}). Proof. Choose some h = h_{N} = (i_{1}, a′_{1}, ..., i_{N}) with histories h_{n} = (i_{1}, a′_{1}, ..., i_{n}), ω = (i_{1}, ..., i_{N}), ω_{n} = (i_{1}, ..., i_{n}). Then by Proposition 25 V̂_{N}(h) − V̂_{0}(i_{0}) = ∑_{n=0}^{N−1}[V̂_{n+1}(h_{n+1}) − V̂_{n}(h_{n})]

≤ ∑_{n=0}^{N−1} ϕ̄_{n}(h_{n}, a′_{n})^{⊤} · ρ_{n+1}(i_{n}, i_{n+1}) where

V̂_{N}(h) = X̆(h), V̂_{0}(i_{0}) = x̄ by (8.6) and (8.8).

Thus we have

X̆(h) ≤ x̄ + ∑_{n=0}^{N−1} ϕ̄_{n}(h_{n}, a′_{n})^{⊤} · ρ_{n+1}(i_{n}, i_{n+1}) for h ∈ H_{N} which implies (8.12) X^{δ}(ω) ≤ x̄ + ∑_{n=0}^{N−1} ϕ̄_{n}^{δ}(ω_{n})^{⊤} · ρ_{n+1}(i_{n}, i_{n+1}) = X_{N}^{ϕ(δ)}(x̄). [] Lemma 24 can be looked upon as a supermartingale property under each martingale measure which was derived by means of dynamic programming in the same sense as in El Karoui & Quenez [7]. One can write (8.12) as (8.13) X^{δ}(ω) = X_{N}^{ϕ(δ)}(x̄) + ∑_{n=1}^{N} c_{n}^{δ} where

c_{n+1}^{δ} := [V̂_{n+1}(h_{n+1}^{δ}(ω_{n+1})) − V̂_{n}(h_{n}^{δ}(ω_{n}))] − ϕ̄_{n}^{δ}(ω_{n})^{⊤} · ρ_{n+1}(i_{n}, i_{n+1}) ≥ 0 . Then c_{n}^{δ} can be interpreted as a possible consumption at time n when hedging X̆. Theorem 26 can be imbedded in the framework of a general optional decomposition theorem for supermartingales under each martingale measure. (Föllmer & Kabanov [10], Föllmer & Kramkov [11], Kramkov [24], Schäl [37]). The analogy between the set of all policies in Markov decision model and the set of martingale measures can also be used to show that Markovian martingale measures are sufficient in the same sense as Markovian policies are sufficient in a Markovian environment ([37]). Acknowledgement. I would like to thank the referee for very useful remarks. Morover, it is a pleasure to acknowledge the influence of my teacher Karl Hinderer who laid much of the foundation of the general non-Markovian theory of dynamic programming. Modern finance is another important area where the non-Markovian setting is natural. In particular, contingent claims of so-called exotic options provide interesting examples of general historydependent reward and cost functions, respectively.

Bibliography

[1] D. Bertsekas, “Necessary and sufficient conditions for existence of an optimal portfolio”, J. Econ. Theory 8, 235-247, 1974.

[2] H. Bühlmann, F. Delbaen, P. Embrechts and A. Shiryaev, “On Esscher transforms in discrete finance models”, Astin Bulletin 28, 171 — 186, 1998.

[3] R.C. Dalang, A. Morton and W. Willinger, “Equivalent martingale measures and noarbitrage in stochastic securities market models”,

Stochastics and Stochastic Reports 29, 185 — 201, 1990.

[4] M.H.A. Davis, “Option pricing in incomplete markets”, in: Mathematics of Derivative Securities, ed: M.A.H. Dempster, S.R. Pliska, Cambridge Univ. Press, 216 — 226, 1997.

[5] F. Delbaen, and M. Yor, “Passport options”, Working paper, ETH Zürich, 1999.

[6] E.B. Dynkin and A.A. Yushkevich, Controlled Markov Processes, Springer, Berlin, 1997.

[7] N. El Karoui and M.C. Quenez, “Dynamic programming and pricing of contingent claims in an incomplete market”, SIAM J. Control Optim. 33, 29-66, 1995.

[8] A.G. Fakeev, “Optimal stopping rules for processes with continuous parameter”, Theory Probab. Appl. 15, 324-331, 1970.

[9] E.A. Feinberg, “Total reward criteria”, This volume, 2000.

[10] H. Föllmer and Yu.M. Kabanov, “Optional decomposition and Lagrange multipliers”, Finance Stochast. 2, 69 — 81, 1998.

[11] H. Föllmer and D. Kramkov, “Optional decompositions under constraints”, Probab. Theory Relat. Fields 109, 1 — 25, 1997.

[12] H. Föllmer and M. Schweizer, “Hedging of contingent claims under incomplete information”, in: Applied Stochastic Analysis, eds. M.H.A. Davis and R.J. Elliot, Gordon and Breach, London, 1990.

[13] M. Fritelli, “The minimal entropy martingale measure and the valuation problem in incomplete markets”, Mathematical Finance. To appear .

[14] P. Grandits, “The p-optimal martingale measure and its asymptotic relation with the minimal-entropy martingale measure”, Bernoulli 5, 225-248, 1999.

[15] P. Grandits and Th. Rheinlaender “On the minimal entropy martingale measure”, Preprint, Techn. Univ. Berlin, Fb. Mathem. 1999.

[16] N. H. Hakansson, “Optimal entrepreneurial decisions in a completely stochastic environment”, Management Science 17, 427-449, 1971.

[16] J.M. Harrison and D.M. Kreps, “Martingales and arbitrage in multiperiod securities markets”, J. Economic Theory 20, 381-408, 1979.

[17] J.M. Harrison and S.R. Pliska, “Martingales and stochastic integrals in the theory of continuous trading”, Stoch. Processes & Appl. 11, 215-260, 1981.

[18] K. Hinderer, Foundations of non-stationary dynamic programming with discrete timeparameter, Lecture Notes in Operations Research and Mathematical Systems 33 Berlin-Heidelberg-New York: Springer, 1970.

[19] A. Hordijk, Dynamic Programming and Markov Potential Theory, Amsterdam: Mathematical Centre Tracts 51, 1974.

[20] J. Jacod and A. N. Shiryaev, “Local martingales and the fundamental asset pricing theorems in the discrete-time case”, Finance Stochast. 3, 259-273, 1998.

[21] I. Karatzas, “On the pricing of American Options”, Appl.Math. Optim. 17, 37-60, 1988.

[22] I. Karatzas and S.G. Kou, “On the pricing of contingent claims under constraints”, Ann. Appl. Probab. 6, 321-369, 1996.

[23] R. Korn and M. Schäl, “On value preserving and growth optimal portfolios”, Math. Methods Op. Res. 50, 189 — 218, 1999.

[24] D.O. Kramkov, “Optional decomposition of supermartingales and hedging of contingent claims in incomplete security markets”,

Probab. Theory Relat. Fields 105, 459 — 479, 1996.

[25] H.-U. Küenle, Stochastische Spiele und Entscheidungsmodelle, Teubner-Texte zur Mathematik, 89. Teubner, Leibzig, 1986.

[26] H. Leland, “On the existence of optimal policies under uncertainty”, J. Econ. Theory 4 , 35-44, 1972.

[27] R.C. Merton, Continuous-time Finance, Blackwell, Cambridge, MA, 1990.

[28] M. Motoczynski, “Multidimensional variance-optimal hedging in discrete time model — a general approach”, Mathematical Finance 10, 243-258, 2000.

[29] M. Motoczynski and L. Stettner, “On option pricing in the multidimensional Cox-Ross-Rubinstein model”, Applicationes Mathematicae 25, 55-72, 1998.

[30] V. Naik and R. Uppal, “Minimum cost hedging in incomplete Markets”, Univ. of British Columbia, Working paper, 1992.

[31] S.R. Pliska, Introduction to Mathematical Finance, Blackwell publisher, Malden, USA, Oxford, UK, 1997.

[32] L.C.G. Rogers, “Equivalent martingale measures and no-arbitrage”, Stochastics and Stochastic Reports 51, 41-49, 1994.

[33] R.T. Rockafeller, Convex Analysis, Princeton Univ. Press, Princeton, N.J, 1970.

[34] W. J. Runggaldier and A. Zaccaria, “A stochastic control approach to risk management under restricted information”, Mathematical Finance 10, 277-288, 2000.

[35] W. Schachermayer, “A Hilbert space proof of the fundamental theorem of asset pricing in finite discrete time”, Insurance: Mathematics and Economics 11, 249 — 257, 1992.

[36] M. Schäl, “On quadratic cost criteria for option hedging”, Math. Oper. Res. 19, 121-131, 1994.

[37] M. Schäl, “Martingale measures and hedging for discrete-time financial markets”, Math. Oper. Res. 24, 509-528, 1999.

[38] M. Schäl, “Portfolio optimization and martingale measures”, Mathematical Finance 10, 289-304, 2000.

[39] M. Schäl, “Price systems constructed by optimal dynamic portfolios”, To be published in Math. Methods Op. Res, 2000.

[40] M. Schweizer, “Variance-optimal hedging in discrete time”, Math. Oper. Res. 20, 1-32, 1995.

[41] M. Schweizer, “Approximation pricing and the variance-optimal martingale measure”, Ann. Prob. 24, 206-236, 1996.

[42] S. E. Shreve and J. V ec̆er̆, “Options on a traded account: vacation calls, vacation puts and passport options”, Finance Stochast. 4, 255-274, 2000.