In the past 40 years, there has been a growing attention in the application of probability in finite groups (for example see [8, 16]). In this paper, we denote by ν(G) the probability that two randomly selected elements of G produce a nilpotent subgroup. In other words we have

The notion ν(G) is introduced in [11] on the model of the commutativity degree, via

Note that for x,y∈G, we have xy=yx if and only if 〈x,y〉 is abelian.

It is easy to see that cp(G)=∑ x∈G|CG(x)||G|2 where CG(x) is the centralizer of x in G as CG(x) is a subgroup of G for any x∈G.

Similarly if

then

However, NilG(x) is not necessarily a subgroup of G, and so it is difficult to glean information about a group G from ν(G).

A finite group G is nilpotent if and only if ν(G)=1 (see Theorem 1 of [5]). On the other hand, Wilson [16] showed that in finite groups G the probability that two random elements of G produce a nilpotent group goes to 0 as the index of the Fitting subgroup of G goes to infinity.

Gustafson [8] proved that if G is a non-abelian group, then cp(G)≤58, and that equality holds if and only if GZ(G) is isomorphic to the Kelian four-group Z2×Z2. Several authors determined the structure of a finite group G when cp(G) is sufficiently large, see [2, 9, 12].

In [7] Guralnick and Wilson found that if G is a nonnilpotent group, then ν(G)≤12. In this paper we classify groups G with ν(G)=12 (see Proposition 2.6).

It is easy to see that cp(A5)=ν(A5)=112 where A₅ is the alternating group of degree five. Dixon observed that cp(G)≤112 for any finite nonabelian simple group G. This was submitted by Dixon as a problem in Canadian Math. Bulletin, 13 (1970), with his own solution appearing in 1973. Guralnick and Robinson [6] extended this result to nonsolvable groups and determined precisely for which nonsolvable groups the equality happens. Recently in [10] the authors of the present paper showed that if G is a group such that NilG(x) is a subgroup of G for every x∈ G and ν(G)>112, then G is solvable.

Fulman et al. [5] proved that if G is a solvable nonnilpotent group and p is the smallest prime number that divides |G|, then ν(G)≤1p and equality holds if and only if p=2 and GZ∞(G) is isomorphic to the dihedral group of order 6 (see [5]). Here Z∞(G) is the hypercenter of G (i.e. the terminal term of the upper central series of G, see [3, 13]). In this article we improve this upper bound as follows.

Theorem 1.1. Suppose that G is a solvable nonnilpotent group of even order. Then ν(G)≤p+34p where p is the smallest odd prime number that divides |G|; equality holds if and only if GZ∞(G(G))≅D2p is the dihedral group of order 2p.

For a prime p we denote by Z_p^k the elementary abelian group of order p^k. We propose the following conjecture for every nonnilpotent group of odd order.

Conjecture Let G be a finite solvable nonnilpotent group such that |G|=p1n1p2n2⋯prnr where 2<p1<⋯<pr are primes. Then

for some 1≤l<k≤r and equality holds if and only if GZ∞(G)≅Zpktk⋊Zpl. We think that this conjecture is true for the class of N-groups, introduced by Abdollahi and Zarrin in [1], which are the groups in which NilG(x) is a nilpotent group for every x∈G∖Z∞(G). We feel that the method used in proof of main theorem of [15] may be useful in proving this.

In Section 2 we compute ν(G) for Frobenius groups and Dihedral groups. We also prove that for any positive integer n, there is a group G such that ν(G)=1n. Finally we classify all groups G with ν(G)=12. In Section 3 we verify Theorem 1.1 and, with Theorem 3.2, confirm the above conjecture for groups of square-free order.

In this article G is a finite group and Z∞(G) is its hypercenter. Most notation we use is standard and follows [14].

The following lemmas are very useful in the sequel.

Lemma 2.1. Suppose that G is a group. Then ν(G)=ν(GZ∞(G)).

Proof. See Corollary 3 of [5].

Lemma 2.2. Suppose that G and H are finite groups. Then ν(G×H)=ν(G)×ν(H).

Proof. The proof is not complicated.

Proposition 2.3. If G=H⋉K is a Frobenius group where the Frobenius kernel is K and the complement is H, then ν(G)=1|H|2(1−1|K|)+ν(H)|K|.

Proof. By hypothesis, we have CG(h)⊆H for each 1≠h∈H, CG(k)⊆K for each 1≠k∈K and H∩Hx=1 for each x∈G∖H. Now if 〈h1k1,h2k2〉 is nilpotent such that h1,h2∈H and k1,k2∈K, then h₁=h₂=1 or k₁=k₂=1. On the other hand {K,(Hx−1)|x∈K} is a partition of G and since K is nilpotent, we are done.

Corollary 2.4. Suppose that G is the dihedral group of order 2^rn where r>1 and n is odd. Then ν(G)=n+34n.

Proof. Since D2rnZ(D2rn)≅D2r−1n for r>1, by Lemma 2.1 we conclude that ν(D2rn)=ν(D2r−1n)=⋯=ν(D2n). Since n is odd, D_2n is a Frobenious group with the cyclic kernel of order n and so we are done by Proposition 2.3.

Corollary 2.5. For any integer n>0, there is a group G of even order such that ν(G)=1n.

Proof. Our proof is by induction on n. If n∈{1,2,3}, then the result holds since ν(D2)=1, ν(D6)=12 and ν(D18)=13. So assume that n≥4 and the that the result holds for all positive integers m<n. If n is even, then there is a group H where ν(H)=2n by induction hypothesis and so ν(H×D6)=1n by Lemma 2.2. Suppose that n is odd, then n=4m+1 or n=4m+3 for some positive integer m. It follows from Corollary 2.4 that ν(D2(4m+1))=m+1n and ν(D2(12m+9))=m+1n. Since m+1<n, we are done by induction hypothesis and Lemma 2.2. This completes the proof.

In the following we classify all groups G with ν(G)=12.

Proposition 2.6. Suppose that G is a finite group (not necessarily solvable). Then ν(G)=12 if and only if GZ∞(G)≅D6, the dihedral group of order 6.

Proof. We get necessity by By Lemma 2.1. Conversely if ν(G)=12, then the probability of solvability of G is equal or greater than 12 and so G is solvable by [7]. By Theorem 5 of [5], we conclude that ν(G)≤12 and equality holds when GZ∞(G)≅D6, as needed.

S. Franciosi and F. Giovanni defined and studied a JNN group as a group all of whose proper quotients are nilpotent (see [4] and [5], Definition 1). It should be noted that a finite group G is a JNN group if and only if G=L⋉ A where A is an elementary abelian p-group and L is a nilpotent group such that p dose not divide the order of G and the action of L on A is faithful and irreducible (See Theorem 4 of [5] and what follows it).

Proof of Theorem 1.1.

If p=3, then ν(G)=1−ν0(G)≤12 by Theorem 5 of [5] and equality holds if and only if GZ∞(G)≅D6. So we assume that the smallest odd prime divisor of |G| is greater than 3. It is enough to prove the result for JNN groups. For if G is a counterexample of minimal order, then there is a nontrivial normal subgroup K of G such that GK is nonnilpotent (since G is solvable). Suppose that r is the smallest odd prime that divides |GK|. If GK is of even order, then ν(G)≤ν(GK)≤r+34r≤p+34p because r≥p. Also if GK is of odd order, then by Theorem 5 of [5], ν(G)≤ν(GK)≤1r≤14≤p+34p which gives a contradiction. So let us assume that G is a JNN group. Then G=L⋉A where L≅Pk×Pk−1×⋯×P1, P_i's are the unique Sylow p_i-subgroups of L and A is an elementary abelian q-group. By setting N=Pk−1⋉⋯⋉(P1⋉A) we have G=P_k⋉ N. We claim that if q=2 and 1≠xp∈Pk, then |CG(xp)∩N|≤|N4|.

Assume that |A|=2t (t≥2) and H=CG(xp)∩A. If |H|=2t−1 and a∈A∖H, then axp=ah1 for some h1∈H. Hence axp2=axph1=ah12=a and so bxp2=b for all b∈ A. But P_k acts faithfully on A which implies that xp2=1, obviously absurd. Hence |H|≤2t−2. If M:=Pk−1×Pk−2×⋯×P1, then CG(xp)∩N=M(CG(xp)∩A)=MH and so |CG(xp)∩N|=|M||H|=|N||A||H|≤|N|4, as claimed.

Now we want to count the ordered pairs (x, y) in a fixed pair (a1N,a2N) for some a1,a2∈G where 〈x,y〉 is nilpotent. By page 14 of [5], the probability that a selected pair (x,y) from the coset pair (xpN,ypN) generates a nilpotent subgroup is not greater than |CG(xp)∩CG(yp)∩N||N| and by our claim this probability is equal or less than 14.

Now we continue by induction on the number k of prime divisors of |L|. Here our aim is showing that if the upper bound mentioned in the assertion is correct for N, it is correct for G too. As mentioned above if q=2, then there is nothing to prove. So assume that q≠ 2. Since the action of P_k on A is faithful, in a similar way it can be seen that |CG(xp)∩N|≤|N|q. If q≥ 5, then this probability is equal or less than 15≤14≤p+34p and by the assumption on N, we conclude that ν(G)≤p+34p. Also it is not hard to see that if k≥ 2, then the equality does not hold, since in this case N is not an elementary group and as mentioned above above in both cases, whether q is equal to 2 or not, the probability is less than 14<p+34p. So it is enough to prove it for the base step of the induction. Assume that G=R⋉A where A=(Zq)n, R is a Sylow r-subgroup and |R|=rm. Then we investigate two cases:

Case 1: Assume that q=2. Then ν(G)≤22n+(22nr2m−22n)×1422nr2m. So ν(G)≤r2m+34r2m<r+34r. As one can see, equality cannot hold in this case.

Case 2: Suppose that r=2. Then ν(G)≤q2n+(q2n22m−q2n)×1qq2n22m=22m+q−1q22m and since q≠ 3, we have 22m+q−1q22m≤q+34q and equality holds if and only if m=1 and hence G≅Z2⋉(Zq)n. Now we claim that n=1.

Let 1≠a∈A and 1≠x∈R. If a^x=a, then 〈ar〉=〈a〉 and since the action of R on A is irreducible, we have 〈a〉=A. Henceforth G≅Z2⋉Zq≅D2q. Otherwise, it can be assumed that CG(R)∩A=1, which results that G≅Z2⋉(Zq)n is a Frobenius group. It follows that ν(G)=qn+34qn (see Proposition 2.1). This implies that the equality exists in our assertion if and only if n=1 and G≅D2q, while G is a JNN group.

Now if G is not a JNN group, so there is a normal subgroup N of G such that GN is a JNN because G is solvable. Let ν(G)=p+34p where p is the smallest odd prime that divides |G| and GN is of even order and p_s be the smallest odd prime that divides the order of GN. Then p+34p=ν(G)≤ν(GN)≤ps+34ps which implies that p=p_s and GN≅D2p. Now by an argument similar to that on page 16 of [5] it can be proved that GZ∞(G)≅D2p. Let GN be of odd order and ps>3 be its smallest prime divisor. Then p+34p=ν(G)≤ν(GN)≤1ps, our final contradiction.

For an odd prime p, we denote by Gp the set of all solvable nonnilpotent groups G of even order such that p is the smallest odd prime that divides the order of G.

Corollary 3.1. Suppose that G∈Gp where p is an odd prime . Then ν(G) is the largest value of ν on Gp if and only if GZ∞(G)≅D2p.

Theorem 3.2. Suppose that G is a finite group of odd order and |G|=p1p2⋯pr where p1<⋯<pr are primes. Then we have

for some 1≤l<k≤r and the equality holds if and only if GZ∞(G)≅Zpk⋊Zpl.

Proof. Similar to the proof of Theorem 1.1 we prove it for JNN groups. Let G=L⋉A be a JNN group and let p1<p2<⋯<pr. Then A≅Zpr and p1p2⋯pr−1|pr−1 since L acts faithfully on A. We proceed by induction as it was done in the Theorem 1.1. Thus set N≤G such that |N|=p2p3⋯pr. It follows that G=P1⋉N. We claim that if the assertion is correct for N it will be correct for G too. It is not hard to see that the probability that a pair selected from the coset pair (xpN,ypN) for some xp,yp∈P1 generates a nilpotent subgroup of G is bounded by |CG(xp)∩CG(yp)∩N||N|. But the action of P₁ on A is faithful and then if both x_p and y_p are not identity, then |CG(xp)∩CG(yp)∩N||N|≤1pr. Now since 1pr<max{pi+pj2−1pipj2|pj|pi−1,1≤i,j≤r}, one can conclude that the bound is right and the equality does not hold when r≥3. Coming back to the base of induction, let |G|=p1p2 with p1<p2. Then G=Zp1⋉Zp2 and ν(G)=p2+p12−1p2p12, as wanted.