It is known that for self-gravitating radiation stars in anti-de Sitter (AdS) spacetime, there is a critical dimension, larger than it, the stars always maintain stability with any central energy density ρc; smaller than it, there is a maximal mass for the ρc and when the ρc continues to increase, the total mass of stars becomes a function of the ρc, and the function appears as an oscillation behavior and therefore the stars become unstable. In this paper we extend this study to the nonlinear case, in this case the equation of state is P=aρb, in AdS spacetime, where a and b are two constant parameters. For the nonlinear case,the equations of gravitational field are more complicated, so the relation of total mass to the central energy density is numerically investigated. In particular, the effect of the two parameters a and b and the relation between spacetime dimension d, mass and ρc are analyzed. We find that the critical dimension changes when the parameters a and b vary, namely the equation of state. In particular, the critical dimension decreases when b decreases.