FINITE MULTIPLICITY, INVARIANT DIFFERENTIAL

OPERATORS AND THE ORBIT METHOD

Ronald L. Lipsman

Department of Mathematics University of Maryland College Park, Maryland 20742 USA

1. Introduction

In this paper we continue to explore the conjecture posed in [13]. Let G be a real algebraic group and H ⊂ G a (Zariski-)closed algebraic subgroup. Assume G/H is connected. Then the conjecture in section 5 of [13]—which I shall refer to as CMC for the Corwin Memorial Conference—speculates that

CMC Conjecture. The following are equivalent.

(i) The quasi-regular representation τ = τ_{G,H} = Ind_{H}^{G}1 has bounded finite multiplicity.

(ii) The algebra D(G/H) of G-invariant differential operators on G/H is commutative. (iii) For generic φ ∈ 𝔥^{⊥}, H has open orbits on G · φ ∩ 𝔥^{⊥}.

In (iii) above, 𝔤 and 𝔥 are the Lie algebras of G and H, 𝔤^{∗} is the real linear dual of 𝔤, and 𝔥^{⊥} = {φ ∈ 𝔤^{∗}: φ(𝔥) = 0}.

Extensive motivation and evidence for the conjecture is offered in [13] (and in several other sources—see below). Let us quickly recall some of it. First of all, strong evidence is offered for the assertion that when G and H are algebraic, then τ (which is of course type I) manifests only three possibilities for the multiplicity function n_{π} in its unique direct integral decomposition τ = ∫ ⊕ ; Ĝ n_{π}π dμ(π)—namely, the multiplicity n_{π} is finite and bounded, finite but unbounded, or identically infinite—of course, μ − a.e. in each case. (μ is a representative of the Plancherel measure class of the homogeneous space G/H.) The CMC conjecture postulates two equivalent criteria for the first eventuality.

Next, we cite some of the afore-mentioned evidence. Specifically, if G and H are nilpotent, (i) and (iii) are equivalent [2], (i)⇒(ii) [3], and (ii)⇒(i) is widely believed, although as yet unproven. If G/H is a symmetric space, conditions (i) and (iii) are true [14] (provided they are true—as expected—when G is reductive); and (ii) has been verified in many cases [13]. If G is exponential solvable, then again (i) and (iii) are equivalent [10][11], and (i)⇒(ii) is proven in [13, Prop.1] (with some additional conditions). Note that a violation of (iii)⇒(i) for non-algebraic G is given by the well-known Grélaud example [13,p. 31]. If G is compact and connected, then the CMC conjecture is proven in [4]. Also, some interesting examples in which the conjecture is true for G semisimple are presented in [13,Expl.5] (see also [15]). Finally, it is worth mentioning [1], a generalization of which will serve as the primary context for this paper. In fact, the category of homogeneous space that we shall investigate here is

Supported by NSF # DMS-92-25063 .

(1.1) G/H, where G = H ⋉ N, N Heisenberg of dimension 2r + 1 and H ⊂ Sp(r, ℝ),

for several different types of groups H. In [1] the conjecture is proven for homogeneous spaces (1.1) with H compact. We will test the truth of the CMC conjecture for spaces (1.1), but drop compactness on H. We will examine two basic examples as well as generalizations of each. The basic examples are H unipotent and H abelian. The generalizations are: U ⊂ H ⊂ P where P is a parabolic subgroup of Sp(r, ℝ) with U as its unipotent radical, and H ⊂ G, an algebraic subgroup of full rank.

The paper is organized as follows. In §3 we consider unipotent subgroups. The main result is Theorem 3.1 and its proof consists largely of an explicit computation and examination of the Weil representation on the unipotent radical of a maximal parabolic subgroup of Sp(r, ℝ). Theorem 3.1 should be taken as further evidence for the truth of the unproven implication (ii)⇒(i) in the general nilpotent case. In §4 we treat abelian semisimple subgroups. The main result is Theorem 4.1, and once again an explicit computation of the Weil representation—this time on Cartan subgroups—is the primary vehicle. In §2 we prove some general results on the conjecture applying to a pair of groups G = H ⋉ N, G_{1} = H_{1} ⋉ N with H_{1} ⊃ H. Then, anticipating the results in §§3 and 4, we apply these general results to obtain the conjecture in the categories described at the end of the preceding paragraph. The precise statements of the results are in Theorems 2.6 and 2.8. Finally, in §5, we present a few possibilities of other homogeneous spaces for which further analysis is likely to lead to proofs of the CMC conjecture in those instances. We close this introduction with the remark that the arguments in §3, are very representative of how the proof of the CMC conjecture might proceed in general (or at least for more general spaces than those encountered in this paper), namely, through a judicious combination of semisimple structure theory, the Weil representation, the Orbit Method and the Mackey Machine.

2. Ancestral Inheritance

Henceforth we assume G = H ⋉N_{r} where N_{r} is a Heisenberg group of dimension 2r +1, and H acts on N_{r} by fixing its one-dimensional center Z. Thus H ↪ Sp(N/Z) = Sp(r, ℝ) and we shall assume the latter is an injection. This is no real loss of generality since the kernel of the map is characteristic and divides out without affecting any of the three conditions in the CMC conjecture. Thus H ⊂ Sp(r, ℝ). We assume H is algebraic, although not necessarily connected. But note that G/H is connected. We think of N_{r} as fixed and H as varying. To emphasize that perspective, we write G_{H} = H ⋉ N_{r}.

Now in this section we shall examine how the truth of any of the conditions in the CMC conjecture for a homogeneous space G_{H}/H of type (1.1) affects the corresponding condition for a homogeneous space G_{H_{1}}/H_{1} when H_{1} ⊃ H. We begin with a detailed examination of the representation theory of G_{H}. The irreducible unitary representations of G_{H} fall into three types:

(a) those trivial on N_{r}. They are parameterized by hn → σ(h), σ ∈ Ĥ. These representations are called completely degenerate.

(b) those trivial on Z. These are representations of H ⋉V, V = N_{r}/Z, a vector group. They are parameterized by a fiber space whose base is V̂ /H and whose fibers are Ĥ_{λ}, λ ∈ V̂: π = π_{σ,λ} = Ind_{H_{λ}N_{r}}^{G}σλ, λ ∈ V̂ = {λ ∈ N̂_{r}, λ|_{Z} = 1}, σ ∈ Ĥ_{λ}. These representations are called degenerate. As an aside we remark that the representations that appear in Ind_{H}^{HV} 1 are precisely π_{λ} := π_{1,λ}.

(c) the generic representations. These lie over the generic (infinite-dimensional irreducible unitary) representations γ_{χ} of N_{r}. The latter are uniquely parameterized by a non-trivial character χ of Z. (Sometimes we fix a non-zero element Z_{0} ∈ 𝔷 so that we can parameterize the characters χ by χ_{t}(exp uZ_{0}) = e^{iut}, u ∈ ℝ, t ≠ 0.) The action of Sp(r, ℝ) on N_{r} preserves the class of γ_{χ}, for any χ, but γ_{χ} does not extend to an ordinary representation of Sp(r, ℝ). In fact, it is very well-known that γ_{χ} extends to an ordinary representation γ̃_{χ} of the two-fold metaplectic covering

group Mp(r, ℝ) = { \atop Sp(r, ℝ)}. Moreover, the unitary equivalence class of γ̃_{χ} only depends on sgn(χ). Let ϵ be the non-trivial central element in the kernel of the cover Mp(r, ℝ) → Sp(r, ℝ). Then, for H ⊂ Sp(r, ℝ), γ̃_{χ} will drop down from H̃, the pullback of H in Mp(r, ℝ), to H precisely when there is a genuine character of H̃, that is a character η of H̃ which equals −1 on ϵ. That will be the case for example if H is compact, unipotent or any semisimple abelian subgroup. These are exactly the cases with which we will be concerned, so usually we can ignore the covering groups. Finally, the generic representations are π_{ν,χ} = ν ⊗ γ̃_{χ} × γ_{χ},

where χ ≠ 1, γ_{χ} ∈ N̂_{r}, and ν ∈ ˆ H̃_{ϵ} = {ν ∈ ˆ H̃: ν(ϵ) = −I}. Of course, if η exists on H̃, then both γ̃_{χ} and ν drop to representations of H. We state our results for the drop down case henceforth.

Now return to the general situation G = H ⋉ N_{r}. The spectrum of the quasi-regular representation τ = τ_{G,H} = Ind_{H}^{G}1 is well-known [8, Thm.7.1]:

Proposition 2.1. If γ̃_{χ} = ∫ ⊕ ; Ĥ n_{χ}(ν)ν dμ_{χ}(ν), then

Ind_{H}^{G}1 = ∫_{−∞}^{⊕∞} ∫ ⊕ ; Ĥ n_{χ}(ν)π_{ν̄,χ} dμ_{χ}(ν)dχ.

Since the class of γ̃_{χ} only depends on sgn(χ), we can state the result even more precisely as follows.

Proposition 2.2. Let γ̃^{±} = ∫^{⊕} n^{±}(ν)ν dμ^{±}(ν). Then

Ind_{H}^{G}1 = ∫_{χ>0}^{⊕} ∫ ^{⊕} n^{+}(ν)π_{ν̄,χ} dμ^{+}(ν)dχ + ∫_{χ<0}^{⊕} ∫ ^{⊕} n^{−}(ν)π_{ν̄,χ} dμ^{−}(ν)dχ.

This leads us immediately to

Corollary 2.3. The multiplicity of the quasi-regular representation τ_{G,H} is exactly the multiplicity of the Weil representation γ̃, meaning that n_{π_{ν,χ}} = n^{sgn(χ)}(ν̄).

To emphasize the role of H, and to deemphasize the role of sgn(χ), we shall write γ̃_{H} for the representation γ̃^{±} on H. Now here is the key result.

Proposition 2.4. Suppose that any of the CMC properties are true for G_{H}/H, G_{H} = H⋉N_{r}, and H_{1} ⊃ H. Then the corresponding property is true for G_{H_{1}}/H_{1}, G_{H_{1}} = H_{1}⋉N_{r}.

Proof. We shall demonstrate each of the three properties individually.

(i) This is easy since the multiplicity of τ_{G,H} is that of γ̃_{H}, and clearly γ̃_{H_{1}}|_{H} = γ̃_{H}. So if the latter has bounded finite multiplicity, so must the former. (This is because restriction and direct integrals commute.)

(ii) The manifold G_{H}/H is diffeomorphic to N = N_{r} The left action of G_{H} on N is comprised of the left action of N and conjugation by H. Therefore, we have an isomorphism D(G_{H}/H) = U(𝔫)^{H}, of the G_{H}-invariant differential operators on G_{H}/H with the H-invariants in the enveloping algebra of N. But clearly U(𝔫)^{H_{1}} ⊂ U(𝔫)^{H}. Thus the commutativity of the latter implies the same for the former.

(iii) This part is a little more difficult than the preceding two. We will actually state and prove the result as a separate (somewhat more general)

Proposition 2.5. Let N be simply connected nilpotent. For A ⊂ Aut N, set G_{A} = A⋉N. Then if A ⊂ B ⊂ Aut N and φ ∈ 𝔟^{⊥} ⊂ 𝔞^{⊥}, we have

#(G_{B} · φ ∩ 𝔟^{⊥})/B ≤ #(G_{A} · φ ∩ 𝔞^{⊥})/A.

Proof. The perps in the notation refer to the superset 𝔫^{∗}, i.e. 𝔟^{⊥} = {φ ∈ 𝔫^{∗}: φ(𝔟) = 0} ⊂ 𝔞^{⊥} = {φ ∈ 𝔫^{∗}: φ(𝔞) = 0}. Now clearly G_{A}·φ∩𝔞^{⊥} = AN_{A}^{φ}·φ where N_{A}^{φ} = {n ∈ N: n·φ(𝔞) = 0}. The implication A ⊂ B ⇒ N_{B}^{φ} ⊂ N_{A}^{φ} is obvious. Next, let {b_{j}n_{j} · φ: j ∈ J} represent the distinct B-orbits on G_{B} · φ ∩ 𝔟^{⊥} = BN_{B}^{φ} · φ; whence b_{j} ∈ B, n_{j} ∈ N_{B}^{φ} ⊂ N_{A}^{φ}. If the inequality were false, then the elements n_{j} · φ ∈ G · φ ∩ 𝔞^{⊥} could not all lie in different A-orbits; in which case an_{i_{1}} · φ = n_{i_{2}} · φ for some a ∈ A and indices i_{1}, i_{2} ∈ J. But then b_{2}an_{i_{1}} · φ = b_{2}n_{i_{2}} · φ ⇒ b_{2}n_{i_{2}} · φ = b_{2}an_{i_{1}} · φ = b_{2}ab_{1}^{−1}b_{1}n_{i_{1}} · φ. That is, b_{1}n_{i_{1}} · φ and b_{2}n_{i_{2}} · φ lie in the same B-orbit, a contradiction. □

Finally, we finish Proposition 2.4 by observing that if H ⊂ H_{1} are algebraic, and the orbits of H on G_{H} · φ ∩ 𝔥^{⊥}, φ ∈ 𝔥^{⊥}, are generically open, then they are finite in number and (so by Proposition 2.5) the same must be true of the H_{1}-orbits on G_{H_{1}} · φ ∩ 𝔥_{1}^{⊥} for generic φ ∈ 𝔥_{1}^{⊥}—from which these are also open.

Note that Proposition 2.5 is valid one functional at a time; there is no ‘genericity’ condition. Now in the next two sections we shall prove the following two results.

Theorem 3.1. Let P ⊂ Sp(r, ℝ) be any maximal parabolic subgroup, U its unipotent radical. Then all three conditions in the CMC conjecture are true for G_{U}/U.

Theorem 4.1. Let C ⊂ Sp(r, ℝ) be any semisimple abelian subgroup. Then all three conditions in the CMC conjecture are true for G_{C}/C iff dim C = r, and all three are false otherwise.

We now combine Proposition 2.4 with these two theorems to obtain rather substantial generalizations of each.

Theorem 2.6. Let P ⊂ Sp(r, ℝ) be any parabolic subgroup, U its unipotent radical. Then all three conditions in the CMC conjecture are true for G_{H}/H, for any algebraic subgroup H satisfying U ⊂ H.

Proof. We may place P inside a maximal parabolic subgroup P_{1}. Then if U_{1} denotes the unipotent radical of P_{1}, we have

U_{1} ⊂ U ⊂ H ⊂ P ⊂ P_{1}.

All three conditions are true for G_{U_{1}}/U_{1}, by Theorem 3.1; hence by Proposition 2.4, they are all true for G_{H}/H. □

Remark 2.7. It is interesting to speculate on the class of subgroups H that are covered by Theorem 2.6. Such an H could not be a proper reductive subgroup. If so, we could select a Cartan involution θ preserving H. But then U and θU would generate the whole group. On the other hand by [6,Thm.30.4], any maximal subgroup is either reductive or a maximal parabolic subgroup. Hence, if we enlarge H to a maximal subgroup, it would have to be a maximal parabolic subgroup P_{1}. The problem is that that new maximal parabolic subgroup P_{1} containing H, might be different from P, the maximal parabolic subgroup that has U as its unipotent radical. Moreover, it is not clear that H must contain U_{1}, the unipotent radical of P_{1}. Nevertheless, I believe it is true that the collection of algebraic subgroups of Sp(r, ℝ) which contain the unipotent radical of some maximal parabolic subgroup coincides with the collection of all algebraic subgroups H that satisfy U ⊂ H ⊂ P for some maximal parabolic subgroup P with unipotent radical U.

Now let H be an algebraic subgroup of G = Sp(r, ℝ). The rank of H is the dimension of a Cartan subgroup of a Levi component of H. We say H has full rank if that number is r. Then we have

Theorem 2.8. Let H ⊂ Sp(r, ℝ) be any algebraic subgroup of full rank. Then all three conditions of the CMC conjecture are true for G_{H}/H.

Like Theorem 2.6, this is an immediate consequence of Proposition 2.4—this time combined with Theorem 4.1. Unfortunately, we cannot say anything about lower rank subgroups. In fact, if r = 2 and H = SU(2), then all three CMC conditions are satisfied. But (again with r = 2) if H is a one-torus, it can be situated in SU(2) so as to give either finite unbounded or uniformly infinite multiplicity [8,Expl.8.vii].

3. Unipotent subgroups

Our task in this section is the proof of

Theorem 3.1. Let P ⊂ Sp(r, ℝ) be any maximal parabolic subgroup, U its unipotent radical. Then all three conditions in the CMC conjecture are true for G_{U}/U.

Proof. The group G_{U} is simply connected nilpotent. Therefore, by the fundamental results of [2], it suffices to prove either of conditions (i) or (iii), since either implies the other two. We will prove condition (i), namely that the quasi-regular representation has bounded finite multiplicity. We shall first demonstrate finite multiplicity for the unipotent radical of a minimal parabolic subgroup, and then deduce the corresponding result for the unipotent radicals stipulated in the theorem.

So let us fix a dimension r and set U = U_{r} the unipotent radical of a minimal parabolic subgroup of Sp(r, ℝ). Then it is very well-known that U can be realized as a semidirect product U = T ⋉ S, where T is the group of unipotent upper triangular real matrices (of dimension r), and S is the group of r × r symmetric real matrices. Note: U has dimension r(r − 1)/2 + r(r + 1)/2 = r^{2}. The group T acts on S by A · B = AB^{t}A. The abelian group S can be identified to its dual as usual via the trace. The elements of Ŝ therefore have a well-defined rank. I shall deduce below, from rather well-known results about the Weil representation on the Siegel maximal parabolic subgroup [5], that the spectrum of the Weil representation on U is concentrated in the irreducible unitary representations that lie over the rank 1 matrices in S. We can parameterize a generic set of these rank 1 matrices by the matrices which have a non-zero real number s in the first row and column and zeroes elsewhere. The stability group in T of such a matrix is the group of unipotent triangular matrices of dimension r − 1. For convenience, we shall denote the stability group by T_{0}. We shall write σ_{s} for the irreducible unitary representation of U obtained by tensoring the character λ_{s} of S determined by s with the trivial representation of T_{0} and then inducing up to U. Then, in fact, we have

Lemma 3.2. The Weil representation γ̃_{U} is given by

(3.3)  γ̃_{U} = 2 ∫_{sgn(χ)s<0}^{⊕} σ_{s} ds.

This result was suggested to me by my colleague Jianshu Li. But a thorough search turned up no reference in the literature. I will supply a proof later in this section.

Now we know (by Corollary 2.3) that the multiplicity of the quasi-regular representation Ind_{H}^{G}1—when G = H ⋉ N, N Heisenberg, H fixing the center of N—is the same as the multiplicity of the Weil representation γ̃_{H}. Hence to complete the proof of Theorem 3.1, all we have to do is show that, for any unipotent radical U_{P} of any maximal parabolic subgroup P in Sp(r, ℝ), the restriction γ̃_{U_{P}} still has bounded finite multiplicity. We shall deduce that by means of a very close examination of the root structure of Sp(r, ℝ). We summarize the facts. The r simple roots are comprised of r − 1 short roots α_{1}, …, α_{r−1} and a long root α_{r}. Here is a listing of all the non-zero roots—broken into two collections, according to whether the long root appears or not:

α_{1}, α_{2}, …, α_{r−1}

α_{1} + α_{2}, α_{2} + α_{3}, …, α_{r−2} + α_{r−1}

⋮

α_{1} + ⋯ + α_{r−1}

− − − − − − − − − − − − − −− α_{r}, α_{r} + α_{r−1}, …, α_{r} + ⋯ + α_{1} α_{r} + 2α_{r−1}, α_{r} + 2α_{r−1} + α_{r−2}, …, α_{r} + 2α_{r−1} + α_{r−2} + ⋯ + α_{1} α_{r} + 2α_{r−1} + 2α_{r−2}, α_{r} + 2α_{r−1} + 2α_{r−2} + α_{r−3}, …, α_{r} + 2α_{r−1} + 2α_{r−2} + ⋯ + α_{1}

⋮

α_{r} + 2α_{r−1} + ⋯ + 2α_{1}.

Note that there are r(r − 1)/2 roots in the first collection, and r(r + 1)/2 in the second. The Lie algebra 𝔲 of U is spanned by the corresponding root spaces, and one can read off the bracket relations from the subscript notation—i.e. two root vectors have non-zero bracket iff the sum of the corresponding roots is a root.

In fact, the root spaces corresponding to the first collection of roots span the Lie algebra of T, and those in the second collection span the Lie algebra of S. Now there are exactly r non-associate classes of maximal parabolic subgroups, and the corresponding unipotent radicals U_{j}, j = 1, …, r, have Lie algebras obtained as follows: 𝔲_{j} = the algebra generated by all root spaces for which the term α_{j} appears in the expression of the root (in terms of the simple roots). We shall set T_{j} = U_{j} ∩ T, S_{j} = U_{j} ∩ S. It is easy to see that U_{j} is a semidirect product U_{j} = T_{j} ⋉ S_{j}. But note that for j = r, the group T_{r} is trivial and U_{r} = S_{r} = S.

So recall that the generic representations appearing in γ̃_{U} are σ_{s} = Ind_{T_{0}S}^{T S}1 × λ_{s}. According to Lemma 3.2, we must examine ∫_{sgn(χ)s<0}^{⊕} σ_{s}|_{U_{j}} ds and demonstrate that it has bounded finite multiplicity. We shall accomplish this by proving the following two lemmas.

Lemma 3.4. The representations σ_{s}|_{U_{j}} are disjoint for distinct s.

Lemma 3.5. For fixed s, the representation σ_{s}|_{U_{j}} has a multiplicity-free decomposition into irreducibles.

Proof of Lemma 3.4. Consider the symmetric matrix W_{0} having a non-zero entry in the first row and column and zero elsewhere—normalized so that λ_{s}(exp tW_{0}) = e^{ist}, t ∈ ℝ. Two key observations are that the element W_{0} is central in 𝔲, and that it lies in every 𝔲_{j}. In fact ℝW_{0} is exactly the center of 𝔲, while the center of 𝔲_{j} will be larger in general. Nevertheless, it is still clear that the central character of any representation that appears in σ_{s} will differ from that of any irreducible constituent of σ_{s′}, if s ≠ s′. Lemma 3.4 follows immediately from that observation.

Proof of Lemma 3.5. Note that it would be enough (for Theorem 3.1) to establish bounded finite multiplicity of σ_{s}|_{U_{j}} independent of s. But in fact we shall prove the restriction is multiplicity-free. We are all set up to use the Subgroup Theorem. Namely,

σ_{s}|_{U_{j}} = (Ind_{T_{0}S}^{T S}1 × λ_{s}) |_{U_{j}},

is an induced representation restricted back to a second subgroup. But we observe that the second subgroup is normal, that is U_{j} ◁ U = T S. This is clear from the fact that if we add any root to another root which contains α_{j} in its presentation, then we either get a non-root or a root which again contains α_{j} in its presentation. So we need to consider the Subgroup Theorem in the instance that the restriction is to a normal subgroup. Let us give a statement of that result.

Theorem 3.6 (Subgroup Theorem for a Normal Subgroup). Let G be locally compact, A and B two closed subgroups with A normal. Suppose that the group AB is closed in G. Then for any character λ of B we have

(Ind_{B}^{G}λ)|_{A} = ∫_{AB\G}^{⊕} Ind_{w·(A∩B)}^{A}λ^{w} dẇ,

where w · g = wgw^{−1} and the character λ^{w} is defined by λ^{w}(wbw^{−1}) = λ(b), b ∈ A ∩ B. The measure dẇ represents the canonical measure class on AB\G.

Proof. This follows from Mackey’s classical Subgroup Theorem, which asserts that—in the absence of normality on A, but with the assumption that the quotient space B\G/A is countably separated—one has

(Ind_{B}^{G}λ)|_{A} = ∫_{B\G/A}^{⊕} Ind_{A∩wBw^{−1}}^{A}λ^{w} dẇ,

Now if A is normal, then set-theoretically we have B\G/A = AB\G, and the closedness of the product group AB guarantees the countable separation. The result is clear.

Now we apply Theorem 3.6 to the current situation A = U_{j} = T_{j}S_{j}, B = T_{0}S inside the group (playing the role of G in Theorem 3.6) U = T S. Furthermore, we have AB = T_{0}T_{j}S_{j} and A∩B = (T_{0} ∩T_{j})S_{j}. The group AB is closed—for example, by the unipotence. Hence, applying Theorem 3.6, we have

(3.7)  σ_{s}|_{U_{j}} = ∫_{T_{0}T_{j}\T}^{⊕} Ind_{g·(T_{0}∩T_{j})S_{j}}^{T_{j}S_{j}}(1 × λ_{s})^{g}.

I claim that the induced representations appearing in the direct integral (3.7) are irreducible and pairwise inequivalent. In fact, λ_{s}|_{S_{j}} is still a non-trivial rank 1 character on S_{j}. Its stability group in U_{j} is precisely T_{0}∩T_{j}. Thus by the Mackey Machine (applied to S_{j}◁T_{j}S_{j}), the representation Ind_{(T_{0}∩T_{j})S_{j}}^{T_{j}S_{j}}1 × λ_{s} is irreducible. Now g ∈ U = T S acts by conjugation (an outer automorphism of T_{j}S_{j}), so any of the representations in the direct integral (3.7) is also irreducible.

Why are they inequivalent? Well, it’s enough to show that g, representing a class in T_{0}T_{j}\T other than the identity coset, moves λ_{s} into a character of S_{j} that is not in its T_{j}-orbit—again from the Mackey Machine (see [7]). But it is clear that g · λ_{s} = t_{j}λ_{s} ⇒ t_{j}^{−1}g ∈ T_{0} ⇒ g ∈ T_{0}T_{j}. This completes the proof of Lemma 3.5.

Note that there is nothing special in the above argument about the case j = r. The application of [7]—to show σ_{s}|_{S} = Ind_{T_{0}S}^{T S}(1 × λ_{s})|_{S} has a multiplicity-free decomposition—is even more basic in that case. Now let us complete the proof of Theorem 3.1 by verifying the spectrum equation (3.3). We invoke the description in [5,§1] of the spectrum of γ̃_{P_{r}}, P_{r} = M_{r}U_{r} = M_{r}S, the Siegel maximal parabolic subgroup with M_{r} = GL(r, ℝ). (Actually, it is described in [5] on the cover of P_{r}, but that will not be relevant as momentarily we restrict to unipotent subgroups where the cover is unnecessary.) The maximal unipotent subgroup U = T S satisfies U ⊂ P_{r} since T ⊂ M_{r}. Now the description of γ̃ = γ̃_{P_{r}} in [5] is of a representation acting on L^{2}(ℝ^{r}), which when restricted to U, yields:

γ̃(t)f(y) = f(t^{−1}y), t ∈ T
γ̃(s_{1})f(y) = λ_{s}(−\frac{1}{2}⟨y, s_{1}y⟩)f(y), s_{1} ∈ S.

(Here, ℝ^{r} arises from T/T_{0}, and ⟨·, ·⟩ denotes the symplectic form on ℝ^{2r} preserved by U.) But it is completely clear from this description that

(3.8)  γ̃_{U}|_{S} = ∫_{ℝ}^{⊕} ∫_{T/T_{0}}^{⊕} t · λ_{s} dṫds,

and that

(3.9)  γ̃_{U}|_{T} = Ind_{T_{0}}^{T}1.

The conclusion follows from the quadratic nature of the multiplier in γ̃(s_{1}), and from results—which originate in Mackey’s development of his “Machine”—in [7] which say: whenever we have a semidirect product MV, V a normal vector group (regularly embedded), then the irreducible unitary representation π_{σ,λ} = Ind_{M_{λ}V}^{MV} σλ, σ ∈ M̂_{λ}, satisfies

(3.10)  π_{σ,λ}|_{V} = dim σ ∫_{M/M_{λ}}^{⊕} m · λ dṁ
(3.11)  π_{σ,λ}|_{M} = Ind_{M_{λ}}^{M}σ.

From these observations, the equation (3.3) is clear. This completes the proof of Lemma 3.2 and with it the proof of Theorem 3.1

Lest one be tempted to speculate at this point that γ̃_{U_{0}} has bounded finite multiplicity for any “reasonable” subgroup U_{0} of U, one need only consider the group T. In fact, by the usual Subgroup Theorem, we have

σ_{s}|_{T} = (Ind_{T_{0}S}^{T S}1 × λ_{s}) |_{T}
= Ind_{T_{0}}^{T}1.

The latter is independent of s, thus the Weil representation γ̃_{T} has uniform infinite multiplicity. Since the implication (ii)⇒(i) is not yet proven in the nilpotent case, it is conceivable that the algebra D(G_{T} /T) could be abelian. We leave the relatively routine check that it is not to the reader.

4. Semisimple abelian subgroups

In this section we prove

Theorem 4.1. Let C ⊂ Sp(r, ℝ) be a semisimple abelian algebraic subgroup. Then all three conditions in the CMC conjecture are true for G_{C}/C iff dim C = r, and all three are false otherwise.

Proof. Any group C as stipulated in the statement of the theorem is contained in a Cartan subgroup of Sp(r, ℝ)—those are the maximal semisimple abelian subgroups. The latter are direct products of the form

C_{j} = A_{j}B_{j}, j = 0, …, r,

where A_{j} = (ℝ^{×})^{j} and B_{j} = 𝕋^{r−j}. The Weil representation γ̃_{C_{j}^{0}} is computed in [9,§3]. (Here, C_{j}^{0} denotes the connected component of the identity.) It has uniform multiplicity 2^{j}. In fact, as in [9] we may select compatible bases of the Lie algebras 𝔫, 𝔞_{j} and 𝔟_{j}. Namely, there are vectors: W_{1}, …, W_{j} ∈ 𝔞_{j}; T_{j+1}, …, T_{r} ∈ 𝔟_{j}; X_{i}, Y_{i} ∈ 𝔫, 1 ≤ i ≤ r satisfying

[X_{i}, Y_{k}] = δ_{ik}Z_{0}
[W_{i}, X_{k}] = δ_{ik}X_{k}, [W_{i}, Y_{k}] = −δ_{ik}Y_{k}
[T_{i}, X_{k}] = δ_{ik}Y_{k}, [T_{i}, Y_{k}] = −δ_{ik}X_{k}
[W_{i}, T_{k}] = 0.

Then D(G_{C_{j}^{0}}/C_{j}^{0}) = U(𝔫)^{C_{j}^{0}} is easily computed to be the free abelian algebra generated by: X_{1}Y_{1}, …, X_{j}Y_{j}, X_{j+1}^{2} + Y_{j+1}^{2}, …, X_{r}^{2} + Y_{r}^{2}, Z_{0}. Also, the orbital condition can be read off from the computations in [9,§4] and [1]. In fact, if one takes a generic linear functional φ ∈ 𝔠_{j}^{⊥} ∼ 𝔫^{∗} of the form

φ = ∑_{i=1}^{r} ξ_{i}X_{i}^{∗} + ∑_{i=1}^{r} η_{i}Y_{i}^{∗} + tZ_{0}^{∗}

with

t ≠ 0, ξ_{i}η_{i} ≠ 0, 1 ≤ i ≤ j, ξ_{k}^{2} + η_{k}^{2} ≠ 0, j + 1 ≤ k ≤ r,

then there are precisely 2^{j} C_{j}^{0}-orbits on G_{C_{j}^{0}} · φ ∩ 𝔠_{j}^{⊥}, j = 0, 1, …, r.

Now any subgroup C of full rank will (up to conjugation) lie between some C_{j} and its neutral component C_{j}^{0}. But in that case: the algebra D(G_{C}/C) is the same as D(G_{C_{j}^{0}}/C_{j}^{0}); the finite index forces the orbits to still be open; and the multiplicity is still finite and bounded. In fact, it follows easily by essentially the same computation as in [9] that the multiplicity is free for C = C_{j}.

Now suppose dim C < r. By reasoning as in the last paragraph, it is enough to consider connected groups C = AB, A ≅ ℝ^{s}, B ≅ 𝕋^{t}, s + t < r. We verify that all three conditions are violated. First, it is clear that the algebras D(G_{C_{j}}/C_{j}) are maximal abelian—adding any another generator from U(𝔫) destroys commutativity. It is also clear from the realization of γ̃_{C_{j}^{0}} in [9], that if s > 0, the multiplicity is infinite. On the other hand, if s = 0, then the multiplicity may be either infinite or finite but unbounded. This can be seen as follows. If T is an algebraic r-torus, then there is no loss of generality in taking γ̃_{T} = ∑_{χ∈T̂^{+}}^{⊕} χ,

where T̂^{+} ≅ {J = (j_{1}, , …, j_{r}) : j_{i} ≥ 0}. Suppose T_{1} ⊂ T is any proper algebraic subtorus. Set T_{1}^{⊥} = {χ ∈ T̂: χ(T_{1}) = 1}. Then restriction defines a natural projection p: T̂ → T̂_{1} with T_{1}^{⊥} as kernel. Furthermore, for any χ_{1} = p(χ), χ ∈ T̂^{+}, the multiplicity of χ_{1} in γ̃_{T_{1}} is #(χ + T_{1}^{⊥} ∩ T̂^{+}). Then the two possibilities are determined by whether the lattice T_{1}^{⊥} intersects the “cone” T̂^{+} in the origin only or in an infinite subcone. In the former case, the translates of T_{1}^{⊥} will intersect the cone in finitely many lattice points—but the number of intersection points will not be bounded over all the translates. Thus, we get unbounded finite multiplicity. In the latter case, the translates of T_{1}^{⊥} will have infinite intersections with the cone and the multiplicity will be uniformly infinite.

Finally, how do we see that the orbit condition is violated? In the case of a Cartan subgroup C_{j}, for generic φ as above, we have—using the notation of section 2—that

G_{C_{j}} · φ ∩ 𝔠_{j}^{⊥} = C_{j}N_{C_{j}}^{φ} · φ.

The dimension of C_{j} · φ is clearly r, and the right-hand side of the displayed equation has the same dimension because the condition n · φ(𝔠_{j}) = 0 eliminates precisely r + 1 variables from N. But when we take a subgroup C ⊂ C_{j} of lower dimension, what happens is that the dimension of C ·φ decreases (to that of dim C generically), while the generic dimension of N_{C}^{φ} · φ increases (because of fewer constraints in the condition n · φ(𝔠) = 0). Thus the orbit condition is violated for semisimple abelian groups of dimension lower than r, and the proof is complete.

5. More general categories.

In this section, I mention some other categories of homogeneous spaces G/H that I have considered (to varying degrees) and that may be amenable to verifying the CMC conjecture.

1. G and H both exponential solvable and algebraic. This situation is not so different from what is considered in this paper since such groups have the structure G = AN, with N simply connected normal and A a vector group acting semisimply. Then we have H = BM, B ⊂ A, M ⊂ N, and homogeneous spaces like this have been studied in [9][13]. The conjecture should follow on the heels of the nilpotent case when that is settled.

2. G and H Strichartz homogeneous spaces. Then G = RV is a semidirect product of a normal vector subgroup V by a (typically) reductive subgroup R, H = SU has the same structure, S ⊂ R, U ⊂ V. These homogeneous spaces have been studied extensively in [12] and [16], and I imagine that all the implications (in the CMC conjecture) except perhaps (ii)⇒(i) should be readily attainable. However, as in [16], some assumptions on the generic stabilizers R_{λ}, λ ∈ U^{⊥}, may be necessary.

3. G semisimple, U ⊂ H ⊂ P, P a parabolic, U its unipotent radical. Some work on the Plancherel theory of these spaces may be found in [15]. The CMC conjecture can be verified if P = MAN is minimal and H is one of the canonical groups N, AN, MN or P itself. More general subgroups await study.

4. G and H both reductive. This is the most interesting and problematic case. One should start perhaps with full rank subgroups, but the general case seems beyond reach at the moment.

References

1. C. Benson, J. Jenkins, R. Lipsman & G. Ratcliff, A geometric criterion for Gelfand pairs associated with the Heisenberg group, Pacific J. Math. (1996) (to appear).

2. L. Corwin, F. Greenleaf & G. Grélaud, Direct integral decompositions and multiplicities for induced representations of nilpotent Lie groups, Trans. Amer. Math. Soc. 304 (1987), 549–583.

3. L. Corwin & F. Greenleaf, Commutativity of invariant differential operators on nilpotent homogeneous spaces G/K with finite multiplicity, Comm. Pure & Appl. Math. 45 (1992), 681–748.

4. V. Guillemin & S. Sternberg, Multiplicity-Free spaces, J. Differential Geometry 19 (1984), 31–56.

5. R. Howe, On a notion of rank for unitary representations of the classical groups, Harmonic Analysis and Group Representations, C.I.M.E. II ciclo 1980, Naples, 1982, pp. 222–332.

6. J. Humphreys, Linear Algebraic Groups, Springer-Verlag, Heidelberg, 1975.

7. A. Kleppner & R. Lipsman, The Plancherel formula for group extensions, Annales Scient. École Norm. Sup. 5 (1972), 459–516; see also ibid, 6 (1973), 103–132.

8. R. Lipsman, Orbital parameters for induced and restricted representations, Trans. Amer. Math. Soc. 313 (1989), 433–473.

9. R. Lipsman, Harmonic analysis on exponential solvable homogeneous spaces, Pacific J. Math. 140 (1989), 117–147.

10. R. Lipsman, Induced representations of completely solvable Lie groups, Annali della Scuola Norm. Pisa 17 (1990), 127–164.

11. R. Lipsman, The multiplicity function on exponential and completely solvable homogeneous spaces, Geometriae Dedicata 39 (1991), 155–161.

12. R. Lipsman, The Plancherel formula for homogeneous spaces with polynomial spectrum, Pacific J. Math. 159 (1993), 351–377.

13. R. Lipsman, Attributes and applications of the Corwin-Greenleaf multiplicity function, Contemporary Math. 177 (1994), 27–46, Representation Theory and Analysis on Homogeneous Spaces, American Mathematical Society, Providence.

14. R. Lipsman, Orbital symmetric spaces and finite multiplicity, J. Functional Analysis 135 (1996), 1–38.

15. R. Lipsman, The Plancherel formula for the horocycle space and generalizations, J. Australian Math. Society (1996) (to appear).

16. R. Lipsman A unified approach to concrete Plancherel theory of homogeneous spaces, (to appear).