Fekete and Szegö proved the remarkable result that the estimate

holds with 0 ≤ λ ≤ 1 for any normalized univalent function

in the open unit disk U. This inequality is sharp for all λ (see [7]). The coefficient functional

in the unit disk represents various geometric quantities. For example, when λ = 1, φ (f(z))=a3-λa22 becomes S_f (0)/6 where S_f denotes the Schwarzian derivative (f″/f′)′ − (f″/f′)²/2. Note that, if we consider the nth root transform [f(zⁿ)]^1/n = z + c_n+1zⁿ⁺¹ + c_2n+1z²ⁿ⁺¹ + … of f with the power series (1.1), then c_n+1 = a₂/n and c2n+1=a3/n+(n-1)a22/2n2, so that

where μ = λn + (n − 1)/2. Moreover, φ (f(z)) behaves well with respect to the rotation, namely φ ((e^−iθf(e^iθz)) = e^2iθφ(f), for θ ∈ R.

This is quite natural to discuss the behavior of φ (f(z)) for subclasses of normalized univalent functions in the unit disk. This is called the Fekete-Szegö problem. Many authors have considered this problem for typical classes of univalent functions (see, for instance [1, 2]).

We denote by A the set of all functions of the from (1.1) that are normalized analytic and univalent in the unit disk U. Also, for 0 ≤ α < 1, let S*(α) and C(α) denote the classes of starlike and convex univalent functions of order α, respectively, i.e., let

and

The notions of α-starlikeness and α-convexity were generalized onto a complex order by Nasr and Aouf [4], Wiatrowski [8], and Nasr and Aouf [3].

Observe that S* (0) = S* and C(0) = C represent standard starlike and convex univalent functions, respectively.

Let f(z)=z+∑k=2∞akzk and g(z)=z+∑k=2∞bkzk be analytic functions in U. The Hadamard product (convolution) of f and g, denoted by f * g is defined by

For a function f(z) in A, we define

and

With the above operator Dⁿ, we say that a function f(z) belonging to A is in the class A(n, m, α) if and only if

for some α (0 ≤ α < 1), and for all z ∈ U.

We note that A(0, 1, α) = S^*(α) is the class of starlike functions of order α, A(1, 1, α) = C(α) is the class of convex functions of order α, and that A(1, 0, α) = A_n(α) is the class of functions defined by Salagean [6].

Definition 1.1

Let b be a nonzero complex number, and let f be an univalent function of the form (1.1), such that D^n+mf(z) ≠ 0 for z ∈ U\ {0}. We say that f belongs to Ψ_n,m(b) if

By giving specific values to n,m and b, we obtain the following important subclasses studied by various researchers in earlier works, for instance, Ψ_1,0(b) = S*(1 − b) (Nasr and Aouf [4]), Ψ_1,1(b) = C(1 − b) (Wiatrowski [8], Nasr and Aouf [3]).

We denote by ℘ class of the analytic functions in U with p(0) = 1 and ℜ(p(z)) > 0. We shall require the following:

Lemma 2.1.([5], p.166)

Let p ∈ ℘ with p(z) = 1+c₁z + c₂z² + …, then

If |c₁| = 2 then p(z) ≡ p₁(z) = (1+γ₁z)/(1−γ₁z) with γ₁ = c₁/2. Conversely, if p(z) ≡ p₁(z) for some |γ₁| = 1, then c₁ = 2γ₁ and |c₁| = 2. Furthermore we have

If |c₁| < 2 and |c2-c122|=2-∣c1∣2, then p(z) ≡ p₂(z), where

and γ₁ = c₁/2, γ2=2c2-c124-∣c1∣2. Conversely if p(z) ≡ p₂(z) for some |γ₁| < 1 and |γ₂| = 1, then γ₁ = c₁/2, γ2=2c2-c124-∣c1∣2 and |c2-c122|=2-∣c1∣2.

Theorem 2.2

Let n,m ≥ 0 and let b be non-zero complex number. If f of the form (1.1) is in Ψ_n,m(b), then

and

Equality in (2.1) holds if D^n+mf(z)/Dⁿf(z) = 1+b [p₁(z) − 1] , and in (2.2) if D^n+mf(z)/Dⁿf(z) = 1+b [p₂(z) − 1] , where p₁, p₂are given in Lemma 2.1.

Remark 2.3

Putting b = 1−α in the above theorem we get the following corollary.

Corollary 2.4

Let n,m ≥ 0 . If f of the form (1.1) is in A(n, m, α), then

and

Remark 2.5

In the above Theorem 2.2 and Corollary 2.4 a special case of Fekete- Szegö problem e.g. for real μ = 2²ⁿ (2^m − 1) /3ⁿ(3^m − 1) occurred very naturally and simple estimate was obtained.

Now, we consider functional ∣a3-μa22∣ for complex μ.

Theorem 2.6

Let b be a non-zero complex number and let f ∈ Ψ_n,m(b). Then for μ ∈ ℂ

For each μ there is a function in Ψ_n,m(b) such that equality holds.

Remark 2.7

Putting b = 1 − α in the above theorem we get the following corollary.

Corollary 2.8

Let f ∈ A(n, m, α) . Then for μ ∈ ℂ

For each μ there is a function in A(n, m, α) such that equality holds.

We next consider the case, when μ and b are real. Then we have:

Theorem 2.9

Let b > 0 and let f ∈ Ψ_n,m(b) . Then for μ ∈ R we have

For each there is a function in Ψ_n,m(b) such that equality holds.

Remark 2.10

Put b = 1 − α in the above theorem we get the following corollary.

Corollary 2.11

Let f ∈ A(n, m, α) . Then for μ ∈ R we have

For each there is a function in A(n, m, α) such that equality holds.