Throughout this article, R will represent an associative ring. Recall that a proper ideal P of R is said to be prime if for any x,y∈R,xRy⊆P implies that x∈ P or y∈ P. Therefore, R is called a prime ring if and only if (0) is the prime ideal of R. R is called a semiprime ring if for any x,y∈ R, xRx=(0) implies that x=0. For any x,y∈ R, the symbol [x,y] will denote the commutator xy-yx, while the symbol x◦ y will stand for the anticommutator xy+yx. A map d:R⇒ R is a derivation of a ring R if d is additive and satisfies d(xy)=d(x)y+xd(y) for all x,y∈ R. A map F:R⇒ R is a generalized derivation of a ring R with d if F is additive and satisfies F(xy)=F(x)y+xd(y) for all x,y∈ R. A map F:R→R is a multiplier of a ring R if F is additive and satisfies F(xy)= F(x)y=xF(y) for all x,y∈ R. During the last two decades, many authors have studied commutativity of prime and semi-prime rings admitting suitably constrained additive mappings acting on appropriate subsets of the rings. Moreover, many of obtained results extend other ones proven previously just for the action of the considered mapping on the entire ring. In this direction, the recent literature contains numerous results on commutativity in prime and semi-prime rings admitting constrained additive mappings, automorphisms, derivations, skew derivations and generalized derivations acting on appropriate subsets of the rings, see [1], [3], [4], [5], [7] and [8].

In the present paper, we adopt a new study, which is an extension and also a generalization of recent results existing in the literature. Precisely, we consider differential identities, in a prime ideal of an arbitrary ring, involving generalized derivation without primeness assumptions on the considered ring.

We first indicate the following lemmas which are essential for developing our paper.

Lemma 2.1. ([7, Proposition 1.3] Let R be a ring, P is a prime ideal of R. If R admits a generalized derivation F with associated derivation d satisfying [x,F(x)]∈P for all x ∈ R, then either R/P is a commutative integral domain or d(R)⊆P.

Lemma 2.2. ([7, Lemma 1.3]) Let R be a ring and P be a prime ideal of R, If

• (i)[x,y]∈ P

• (ii) x◦ y∈ P

for all x,y ∈ R, then R/P is a commutative integral domain.

The following result is a generalization of [5, Lemma 1].

Theorem 2.3. Let R be a ring, P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying F(x)x±xG(x)∈P for all x∈ R, then either g(R)⊆P or R/P is a commutative integral domain.

proof Assume that

(2.1)F(x)x−xG(x)∈P

for all x∈ R. By linearizing (2.1), we have

(2.2)F(x)y+F(y)x−xG(y)−yG(x)∈P

for all x,y∈ R. Replacing y by yx in (2.2), we get

(2.3)F(x)yx+F(y)x2+yd(x)x−xG(y)x−xyg(x)−yxG(x)∈P

for all x,y∈ R. Right multiplying (2.2) by x, we obtain

(2.4)F(x)yx+F(y)x2−xG(y)x−yG(x)x∈P

for all x,y ∈ R. Subtracting (2.4) from (2.3), this gives

(2.5)yd(x)x−xyg(x)−yxG(x)+yG(x)x∈P

for all x,y∈ R. Putting ry instead of y in (2.5), where r ∈ R, we have

(2.6)ryd(x)x−xryg(x)−ryxG(x)+ryG(x)x∈P

for all x,y,r∈ R. Left multiplying (2.5) by x, we get

(2.7)ryd(x)x−rxyg(x)−ryxG(x)+ryG(x)x∈P

for all x,y,r∈R. Comparing (2.6) and (2.7), we obtain [x,r]yg(x)∈P that is [x,r]Rg(x)⊆P. Since P is a prime ideal of R, we get [x,r]∈ P or g(x)∈ P, which implies that I={x∈R | [x,r]∈P} and J={x∈R | g(x)∈P}. But a group cannot be written as the union of two of its proper subgroups then I=R in which case R/P is a commutative integral domain by lemma 2.2 (i) or J=R in this case g(R)⊆P. This completes the proof of the Theorem.

Suppose that F(x)x+xG(x)∈P for all x, y ∈ R, then using the similar arguments as in the above with suitable slight modification, we get the required result.

Corollary 2.4. ([5, Lemma1]) Let R be a ring, P be a prime ideal of R and d and g are derivations. If d(x)x±xg(x)∈P for all x∈ R, then either g(R)⊆P or R/P is a commutative integral domain.

The following result extends [5, Theorem 1 (2)] to its full generalization.

Theorem 2.5. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying F(x)G(y)±[x,y]∈P for all x,y∈ R, then R/P is a commutative integral domain.

Proof. By our hypothesis, we have

(2.8)F(x)G(y)+[x,y]∈P

for all x,y ∈ R. Replacing y by yt in (2.1) and using it, where t ∈ R, we have

(2.9)F(x)yg(t)+y[x,t]∈P

for all x,y,t ∈ R. Putting t=x in (2.2), we get F(x)yg(x)∈P, that is F(x)Rg(x)⊆P. Since P is a prime ideal of R, then F(x)∈P or g(x)∈P, which implies that I={x∈R | F(x)∈P} and J={x∈R | g(x)∈P}. Since a group cannot be the union of its two proper subgroups (abbreviated as Brauer's Trick) either I=R or J=R. If I=R then F(R)⊐ P and by using last relation in (2.1), we obtain [x,y]∈ P this implies that R/P is a commutative integral domain. If J=R, then

(2.10)g(R)⊆P.

By using (2.3) in (2.2), gives y[x,t]∈ P and since P≠ R, then [x,t]∈ P, this implies that R/P is a commutative integral domain.

Assuming that F(x)G(y)−[x,y]∈P for any x, y ∈ R, we can obtain the required result using the same procedures as above with the appropriate changes.

The following Corollary is an immediate consequence of our previous result:

Corollary 2.6. ([5, Theorem 1 (2)]) Let R be a ring and P be a prime ideal of R and d and g are derivations. If d(x)g(y)±[x,y]∈P for all x,y∈ R, then R/P is a commutative integral domain.

Using the same technique as used in the proof of Theorem 2.5 with necessary variations we get the following result.

Theorem 2.7. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying F(x)G(y)±x∘y∈P for all x,y∈ R, then R/P is a commutative integral domain.

A very immediate corollary of Theorem 2.7 is the following result.

Corollary 2.8. Let R be a ring and P be a prime ideal of R and d and g are derivations. If d(x)g(y)±x∘y∈P for all x,y ∈ R, then R/P is a commutative integral domain.

In [2, Theorem 3], Bell and Kappe proved that d = 0 on R if d is a derivation of a prime ring R which acts as homomorphism or anti-homomorphism on a nonzero right ideal of R. Moreover, the first author [6] established this result for generalized derivations of prime rings. He obtained that R must be commutative if R is a 2-torsion free prime ring and F acts as a homomorphism or an anti-homomorphism on a nonzero ideal of R. Recently, in [4, Theorem 4] A. Mamouni et al. studied the behaviour of prime ideal without making any assumption with derivation d satisfying the identities d(xy)−d(x)g(y)∈P for all x,y∈ R or d(xy)−d(y)g(x)∈P for all x,y∈ R, where P is a prime ideal. Our work is then motivated by the previous results. Our objective is to generalize the mentioned result above by replacing the derivation with generalized derivation. Explicitly we shall prove the following theorems:

Theorem 2.9. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying F(xy)±F(x)G(y)∈P for all x,y∈ R, then d(R)⊆P or R/P is a commutative integral domain.

Proof. Assume that

(2.11)F(xy)+F(x)G(y)∈P

for all x,y ∈ R. Replacing y by yr in (2.8) and using it, where r∈ R, we have

(2.12)xyd(r)+F(x)yg(r)∈P

for all x,y,r ∈ R. Writing F(x)y instead of y in (2.12), we get

(2.13)xF(x)yd(r)+F(x)2yg(r)∈P

for all x,y,r ∈ R. Left multiplying (2.5) by F(x), we obtain

(2.14)F(x)xyd(r)+F(x)2yg(r)∈P

for all x,y,r ∈ R. Subtracting (2.13) from (2.14), we have

(2.15)[F(x),x]yd(r)∈P

for all x,y,r∈R, that is [F(x),x]Rd(r)⊆P. Then [F(x),x]∈P or d(r)∈P. In case [F(x),x]∈P, by lemma 2.1, then either d(R)⊆P or R/P is a commutative integral domain. On the other case d(R)⊆P.

Assuming that F(xy)−F(x)G(y)∈P for all x,y∈R, then using the same techniques as used above with necessary variations we get the required result.

Corollary 2.10. ([4, Theorem 4 (1)]) Let R be a ring and P be a prime ideal of R and d and g are derivations. If d(xy)±d(x)g(y)∈P for all x,y∈R, then either d(R)⊆P or R/P is a commutative integral domain.

Theorem 2.11. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying F(xy)±F(y)G(x)∈P for all x,y∈ R, then either R/P is a commutative integral domain or d(R)⊆P.

Proof Assume that

(2.16)F(xy)+F(y)G(x)∈P

for all x,y ∈ R. Replacing x by xy in (2.16) and using it, we have

(2.17)xyd(y)+F(y)xg(y)∈P

for all x,y∈ R. Writing tx instead of x in (2.17), we get

(2.18)txyd(y)+F(y)txg(y)∈P

for all x,y,t∈ R. Left multiplying (2.17) by t, where t∈ R, we obtain

(2.19)txyd(y)+tF(y)xg(y)∈P

for all x,y,t ∈ R. Comparing (2.18) and (2.19), this gives [F(y),t]xg(y)∈P that is [F(y),t]Rg(y)⊆P. Since P is a prime ideal of R, we get [F(y),t]∈ P or g(y)∈ P, which implies that I={y∈R | [F(y),R]⊆P} and J={y∈R | g(y)∈P}. Since R=I∪J, and using again Brauer's Trick, one has that either R=I or R=J. If R=I, then by lemma 2.1, either d(R)⊆P or R/P is a commutative integral domain. On the other hand of J=R, then

(2.20)g(R)⊆P.

By using (2.20) in (2.17), we have xyd(y) ∈ P and since P≠ R, we obtain

(2.21)yd(y)∈P

for all y ∈ R. By linearizing (2.21), we get

(2.22)xd(y)+yd(x)∈P

for all x,y ∈ R. Putting ry instead of y in (2.22), where r∈ R, we obtain

(2.23)xd(r)y+xrd(y)+ryd(x)∈P

for all x,y,r∈ R. Left multiplying (2.22) by r, where r∈ R, we have

(2.24)rxd(y)+ryd(x)∈P

for all x,y,r ∈ R. Subtracting (2.24) from (2.23), we get xd(r)y+[x,r]d(y) ∈ P.

Replacing x by wx in the previous relation, one gets [w, r]xd(y) ∈ P for all x, y, w, r ∈ R. From this, we easily get either R/P is a commutative integral domain or d(R) ⊐ P.

If we have F(xy)−F(y)G(x)∈P for all x,y∈p, then arguing as above, we conclude that either R/P is a commutative integral domain or d(R)⊆P.

Corollary 2.12. ([4, Theorem 4 (2)]) Let R be a ring and P be a prime ideal of R and d and g are derivations. If d(xy)±d(y)g(x)∈P for all x,y ∈ R, then either d(R)⊆P or R/P is a commutative integral domain.

Theorem 2.13. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively. If G(xy)±F(x)F(y)∈P for all x,y∈ R, then d(R)⊆P and g(R)⊆P.

Proof Assume that

(2.25)G(xy)+F(x)F(y)∈P

for all x,y ∈ R. Replacing y by yr in (2.25) and using it, we have

(2.26)xyg(r)+F(x)yd(r)∈P

for all x,y ∈ R. Now, replacing x by zx in (2.26) we find that

(2.27)zxyg(r)+F(z)xyd(r)+zd(x)yd(r))∈P

for all x,y, z, r ∈ R. Letting x=z in (2.26) we get

(2.28)zyg(r)+F(z)yd(r)∈P

for all ,y, z, r∈ R. Again replace y by xy in (2.28), to get

(2.29)zxyg(r)+F(z)xyd(r)∈P.

By using comparing (2.29) in (2.27), we get zd(x)yd(r) ∈ P for all x, y, z, r ∈ R, that is Rd(x)Rd(r)⊆P. Hence

(2.30)d(R)⊆P.

By using (2.30) in (2.29), we have zxyg(r)∈ P and so

(2.31)g(R)⊆P.

Now if G(xy)−F(x)F(y)∈P for all x, y ∈ R, with a slight modification, we get the required result.

Corollary 2.14. Let R be a ring and P be a prime ideal of R and d and g are derivations. If g(xy)±d(x)d(y)∈P for all x,y∈ R, then d(R)⊆P and g(R)⊆P

Theorem 2.15. Let R be a ring and P be a prime ideal of R. If R admits generalized derivations F and G with associated derivations d and g respectively satisfying G(xy)±F(y)F(x)∈P for all x,y∈R, then either R/P is a commutative integral domain

or d(R)⊆P.

Proof Assume that

(2.32)G(xy)+F(y)F(x)∈P

for all x,y ∈ R. Replacing x by xy in (2.32) and using it, we have

(2.33)xyg(y)+F(y)xd(y)∈P

for all x,y ∈ R. Writing tx instead of x in (2.33), we get

(2.34)txyg(y)+F(y)txd(y)∈P

for all x,y,t ∈ R. Left multiplying (2.33) by t, where t ∈ R, we obtain

(2.35)txyg(y)+tF(y)xd(y)∈P

for all x,y,t ∈ R. Comparing (2.34) and (2.35), gives [F(y),t]xd(y) ∈ P that is [F(y),t]Rd(y)⊆P. Since P is a prime ideal of R, we get [F(y),t]∈P or d(y)∈P, which implies that I={y∈R | [F(y),R]⊆P} and J={y∈R | d(y)∈P}. Then I and J are additive subgroups of R whose union is R and using Brauer's Trick, we have either I=R, or R=J. In case I=R, by lemma 2.1, either d(R)⊆P or R/P is a commutative integral domain. In case J=R, d(R)⊆P.

Now if we consider G(xy)- F(y)F(x)∈ P for all x, y ∈ R then the same reasoning proves that either R/P is a commutative integral domain or d(R)⊆P.

Corollary 2.16. Let R be a ring and P be a prime ideal of R and d and g are derivations. If g(xy)±d(y)d(x)∈P for all x,y∈ R, then either d(R)⊆P or R/P is a commutative integral domain.

The following examples show that the condition "primeness of P" in all Theorems cannot be omitted.

Example 2.17. Consider the ring R={0ab002c000:a,b,c∈ℤ4}. Let P={000000000} be an ideal of R

and F=G,d=g:R→R defined by F0ab002c000=02a0000000 with d0ab002c000=00c000000 be generalized derivations. We see that F(x)x−xG(x)∈P, F(x)x+xG(x)∈P, F(xy)−F(x)G(y)∈P, F(xy)+F(x)G(y)∈P, F(xy)−F(y)G(x)∈P, F(xy)+F(y)G(x)∈P, G(xy)−F(x)F(y)∈P, G(xy)+F(x)F(y)∈P, G(xy)−F(y)F(x)∈P and G(xy)+F(y)F(x)∈P. But d(R)⊆P and R/P is noncommutative. Also we see that P is not prime ideal of R because 010000000R010000000⊆P, but 010000000∈P.

Example 2.18. Consider S be a ring such that s²=0 for all s∈ S, but the product of some elements of S is nonzero. Since s²=0, so (s+t)²=0 for all s, t∈ S this implies that s ◦ t=0 for all s, t ∈ S. Suppose

R={xy0x:x,y∈S}. Define F=G=d=g:R→R as Fxy0x=0y00, clearly F and d are (generalized) derivations. Let P={03y00:y∈S}. We see that 0y00R3x003x⊆P, but 0y00,3x003x∈P so P is not prime ideal of R. Also we see that F(x)G(y)+x∘y∈P and F(x)G(y)−x∘y∈P, but d(R)⊆P and R/P is noncommutative.

The authors are greatly indebted to the referee for his/her constructive comments and suggestion, which improves the quality of the paper.