Dual integrable representations of abelian locally compact groups were introduced in [6] (see also [1, 7]). Left regular representations are well known examples of such dual integrable representations. In this paper, we initiate the stucy of this concept for locally compact hypergroups and prove its basic properties. We find some some results which are novel even for the group case. In fact, we give an equivalence condition for a dual integrable representation to have an admissible vector, which is a version of our recent results in [9].

Throughout this paper K is a locally compact hypergroup. For the definitions and basic properties of hypergroups we refer to the monographs [8] (in which hypergroups are called "convo") and [2] (see also [5]). The space of all complex Radon measures on K is denoted by M(K), and the space of positive elements of M(K) is denoted by M+(K). B(H) denotes the space of all bounded operators on a Hilbert space H.

In this section, first we recall the following definition from [8, 11.3].

Definition 2.1. Let H be a Hilbert space. A norm-decreasing *-homomorphism π:M(K)→B(H) is called a representation of K on H if π(δe)=IdH and for each φ,ψ∈H, the mapping μ↦〈π(μ)φ,ψ〉 from M+(K) to ℂ is continuous, where M+(K) is equipped with the cone topology. For each x ∈ K, we denote πx:=π(δx). Let K be a commutative hypergroup with a left Haar measure m.

The left regular representation τ:M(K)→B(L2(K)) is defined by τ(μ)f:=μ∗f, where f∈L2(K) and μ∈M(K). In particular, let H be a compact subgroup of a locally compact group G and G/​​/H be the double coset hypergroup introduced in [8, 8.2] with quotient mapping q:G→G/​​/H defined by q(x):=HxH. Then, G/​​/H has a left Haar measure m [8, 8.2B], and for each f∈L2(G/​​/H,m) the left regular representation of G/​​/H can be obtained by

where σ is the normalized Haar measure on H and μ∈M(G/​​/H).

Definition 2.2. Let K be a commutative hypergroup with a Haar measure ω on K^ [8, 7.3I]. A representation π:M(K)→B(Hπ) is called dual integrable if there exists a corresponding mapping ⋅,⋅π:Hπ×Hπ→L1(K^,ω), called bracket product, such that for all φ,ψ∈Hπ and μ∈M(K),

where μ^ is the Fourier-Stieltjes transform of µ.

Remark 2.3. The bracket product of a dual integrable representation is conjugate symmetry and linear in the first argument. Also, for each x∈ K, putting μ=δx, we have

In particular,

For each φ∈Hπ, φ,φ=0 a.e. if and only if φ=0. In fact, if φ=0, then by (2.1) for each x∈ K we have

and so, by [2, 2.1.6(vi)], φ,φ=0 a.e. The converse follows from (2.2).

Example 2.4. The left regular representation τ of a commutative hypergroup K is dual integrable. In fact, for each φ,ψ∈L2(K) and μ∈M(K) by [2, 2.2.26 and 2.2.34],

with φ,ψτ(ξ):=φ^(ξ)ψ^(ξ)¯.

Remark 2.5. Same as the group case, for each representations (π,Hπ) and (π′,H π ′ ) of a hypergroup K, if π and π' are equivalence via a unitary operator U:Hπ→H π′ and π is dual integrable, then π' is also dual integrable. In fact,

and so, φ′,ψ′π′=U*φ′,U*ψ′π for all φ′,ψ′∈H π ′ , where U^* is adjoint of the operator U.

Proposition 2.6. Let G₁ and G₂ be abelian locally compact groups and π and σ be dual representations of G₁ and G₂ respectively. Then, π⊗σ is a dual integrable representation of G1×G2.

Proof. For each φ1,φ2∈Hπ and ψ1,ψ2∈Hσ we define

Then, by [4, Proposition 4.6] and [4, Section 7.3], for each x ∈ G₁ and y ∈ G₂ we have

and the proof is complete.

Corollary 2.7. Let π be a dual integrable representation of a locally compact abelian group G₁ and τ be the left regular representation of a locally compact abelian group G₂. Then, the induced representation indG1G1×G2 is dual integrable.

Proof. The proof is obtained from Proposition , Example 2.3 and [4, Proposition 7.26].

In spite of the group case, for a commutative hypergroup K we only have

while in general K^⊆Xb(K).

There are hypergroups with K^≠Xb(K) (see [8, Example 9.5]). Although, for a large class of hypergroups, like orbit hypergroups, we have K^=Xb(K) (for more details see [3]). The following theorem holds for hypergroups with K^=Xb(K).

Theorem 2.8. Let K be a commutative hypergroup with Xb(K)=K^ and π be a representation of K on Hπ. Then, there is a projection-valued Borel measure T on K^ such that

Also, for each φ,ψ∈Hπ, there is a complex Borel measure μφ,ψ on K^ such that

for all Borel set E⊆K^, and

Proof. By [8, 6.1G], M(K) is a commutative Banach *-algebra. Therefore, by [4, 1.54], there exists a unique regular projection-valued measure T on K^ such that

for each μ∈M(K). Then, for each φ,ψ∈Hπ,

The mapping μφ,ψ defined by

for all Borel set E⊆K^, is a meaure on K^ and

As the group case, by [2, 2.1.6(vi)], we can conclude:

Corollary 2.9. Let (π,Hπ) be a representation of a commutative hypergroup K and ω be the Haar measure on K^. Then, the following statements are equivalent:

• (i) π is dual integrable.

• (ii) For each φ,ψ∈Hπ, the measure μφ,ψ is absolutely continuous with respect to ω.

Corollary 2.10. Let (π,Hπ) be a dual integrable representation of a commutative hypergroup K. Then, for all φ,ψ∈Hπ:

• (i) φ,φπ≥0 a.e.

• (ii) φ,ψπ≤φ,φπ1/2.ψ,ψπ1/2 a.e.

• (iii) φ,ψπ1≤φψ.

Corollary 2.11. Let (π,Hπ) be a dual integrable representation of K and φ∈Hπ. Put Aφ:={ξ∈K^;φ,φπ(ξ)>0}. Then, for all ψ∈Hπ, φ,ψπ=φ,ψπχAφ in L1(K^).

Proof. By the Cauchy-Schwartz inequality,

and the proof is complete.

Corollary 2.12. Let (π,Hπ) be a dual integrable representation of a commutative hypergroup K. Then, for each μ∈M(K) and φ,ψ∈Hπ, we have

Proof. By dual integrability of π, [2, 2.1.6(iv)] and [8, 12.1E], for all μ,ν∈M(K) and φ,ψ∈Hπ we have

On the other hand,

So,

Finally, by uniqueness in Rieze Representation Theorem and [8, 7.3H],

Similarly,

Definition 2.13. Let π be a dual integrable representation of a commutative hypergroup K on Hπ. For each φ,ψ∈Hπ, we say φ and ψ are π-orthogonal, and write φ⊥πψ, if φ,ψπ=0 a.e. on K^. A subset E⊆Hπ is called π-orthogonal if for each distinct φ,ψ∈E, φ⊥πψ.

Corollary 2.14. Let (π,Hπ) be a dual integrable representation of a Pontrjagin hypergroup K. Then, φ,ψ∈Hπ are π-orthogonal if and only if φ⊥π(M(K))ψ¯ in Hπ.

Proof. Since π is dual integrable, φ⊥π(M(K)ψ¯ if and only if

for all μ∈M(K). Thus, for each x∈ K, setting μ=δx, we have

This implies that φ,ψπ=0 a.e. in K^. Clearly, by (2.3), if φ,ψπ=0 a.e. in K^, then φ⊥π(M(K)ψ¯.

Proposition 2.15. Let π be a dual integrable representation of a commutative hypergroup K on Hπ. The subset E⊆Hπ is π-orthogonal if and only if the set {π(μ)φ:φ∈E,μ∈M(K)} is orthogonal.

Proof. For each φ,ψ∈E and μ,ν∈M(K), by Corollary we have

Hence, if E is π-orthogonal, then π(μ)φ,π(ν)ψ=0 for all distinct φ,ψ∈Hπ and μ,ν∈M(K).

Conversely, if {π(μ)φ:φ∈E,μ∈M(K)} is orthogonal, then putting ν=δe and μ=δx(x∈K) in (5), we have

for all distinct φ and ψ∈Hπ, and so φ,ψ^π=0 a.e. Finally, by [2, 2.1.6(vi)] we have φ,ψπ=0 in L1(K^), and the proof is complete.

Corollary 2.16. Let (π,Hπ) be a dual integrable representation of a commutative hypergroup K and φ,ψ∈Hπ. The following statements are equivalent:

• (i) φ⊥πψ

• (ii) φ⊥π(M(K))ψ¯

• (iii) π(M(K))φ¯⊥π(M(K))ψ¯.

Proposition 2.17. Let (π,Hπ) be a dual integrable representation of a strong commutative compact hypergroup K and φ∈Hπ. If φ,φπ=1 in L1(K^), then the set {πxφ:x∈K} is an orthogonal system in Hπ.

Proof. Since π is dual integrable, for each x,y∈ K we have

Therefore, if φ,φπ=1 in L1(K^), then by [2, proof of Theorem 2.2.9(ii)] and [2, Theorem 2.4.3], πxφ,πyφ=0.

We recall the next definition from [9].

Definition 2.18. Let K be a hypergroup with a (left) Haar measure m, H be a subhypergroup of K, and π:M(K)→B(Hπ) be a representation of K on a Hilbert space Hπ, and V⊆Hπ. A vector h0∈Hπ is called a (π,V)-admissible vector with respect to H if there are constant numbers A,B>0 such that for every h ∈ V,

where m_H is a left Haar measure on H.

If A=B=1, h₀ is called Parseval admissible.

Proposition 2.19. Let K be a Pontrjagin hypergroup with a left Haar measure m, π:M(K)→B(Hπ) be a dual integrable representation of K corresponding to a Plancherel measure ω, and V⊆Hπ. A vector h0∈Hπ is a Parseval (π,V)-admissible vector if and only if for every h∈ V, ‖h‖=‖h0,h‖2.

Proof. Let h0∈Hπ. Since π is dual integrable, for every h∈ V we have

So by Plancherel Theorem, h₀ is a Parseval (π,V)-admissible vector if and only if for each h ∈ V,