On m-tuples of nilpotent $2\times 2$ matrices over an arbitrary field

1. Introduction

1.1. Algebras of invariants

All vector spaces, algebras, and modules are over an arbitrary (possibly finite) field $𝔽$ of arbitrary characteristic $p\ge 0$ unless otherwise stated. By an algebra we always mean an associative algebra with unity.

Consider an n-dimensional vector space V over the field $𝔽$ with a fixed basis $v_1,\dots ,v_n$. The coordinate ring $𝔽[V]=𝔽[x_1,\dots ,x_n]$ of V is isomorphic to the symmetric algebra $S(V^{\ast })$ over the dual space $V^{\ast }$, where $x_1,\dots ,x_n$ is the basis for $V^{\ast }$. Let G be a subgroup of $\mathrm{\text{GL}}(V)\cong {\mathrm{\text{GL}}}_n(𝔽)$. The algebra $𝔽[V]$ becomes a G-module with

for all $f\in V^{\ast }$ and $v\in V$. The algebra of polynomial invariants is defined as follows:

$$𝔽{[V]}^G=\{f\in 𝔽[V]|g\cdot f=f\mathrm{\text{for all}}g\in G\}.$$

The algebra of polynomial functions $\mathcal{𝒪}(V)=𝔽[{\overline{x}}_1,\dots ,{\overline{x}}_n]$, where ${\overline{x}}_i:v\mapsto v_i$ for $v=(v_1,\dots ,v_n)\in V$ with respect to the fixed basis of V. The algebra $\mathcal{𝒪}(V)$ becomes a G-module by formula (1) for all $f\in \mathcal{𝒪}(V)$ and $v\in V$. The algebra of invariant polynomial functions is defined as follows:

Obviously, $\mathcal{𝒪}(V)\simeq 𝔽[V]∕\mathrm{\text{Ker }}(\mathrm{\Psi })$ for the homomorphism of algebras $\mathrm{\Psi }:𝔽[V]\to \mathcal{𝒪}(V)$ defined by $x_i\to {\overline{x}}_i$. In case $𝔽$ is infinite we have that $\mathrm{\text{Ker }}(\mathrm{\Psi })=0$ and $𝔽{[V]}^G=\mathcal{𝒪}{(V)}^G$. If $𝔽={𝔽}_q$ is a finite field, then the ideal $\mathrm{\text{Ker }}(\mathrm{\Psi })$ is generated by $x_i^q-x_i$ for all $1\le i\le n$. Using $\mathrm{\Psi }$ we can consider any element $f\in 𝔽[V]$ as the map $V\to 𝔽$. Namely, we write $f(v)$ for $\mathrm{\Psi }(f)(v)$, where $v\in V$. For any infinite extension $𝔽\subset 𝕂$ we denote $V_{𝕂}=V{\otimes }_{𝔽}𝕂$. Therefore,

Assume that $W\subset V$ is a G-invariant subset of V. The algebra of polynomial functions $\mathcal{𝒪}(W)$ on W is generated by functions $y_1,\dots ,y_n:W\to 𝔽$, where $y_i$ is the restriction of ${\overline{x}}_i:V\to 𝔽$ to W. The algebra $\mathcal{𝒪}(W)$ becomes a G-module by formula (1) for all $f\in \mathcal{𝒪}(W)$ and $v\in W$. Denote by $I(W)$ the ideal of all $f\in \mathcal{𝒪}(W)$ that are zeros over W. Then we have $\mathcal{𝒪}(W)\simeq \mathcal{𝒪}(V)∕I(W)$ and $y_i={\overline{x}}_i+I(W)$. The algebra $\mathcal{𝒪}{(W)}^G$ of invariant polynomial functions on W is defined in the same way as $\mathcal{𝒪}{(V)}^G$. The degree of $f\in \mathcal{𝒪}(W)$ is defined as the minimal degree of a polynomial $h\in 𝔽[V]$ with $f=\mathrm{\Psi }(h)+I(W)$.

1.2. Separating invariants

In 2002, Derksen and Kemper [6] (see [7] for the second edition) introduced the notion of separating invariants as a weaker concept than generating invariants. Given a subset S of $\mathcal{𝒪}{(W)}^G$, we say that elements $u,v$ of W are separated by S if exists an invariant $f\in S$ with $f(u)\ne f(v)$. If $u,v\in W$ are separated by $\mathcal{𝒪}{(W)}^G$, then we simply say that they are separated. A subset $S\subset \mathcal{𝒪}{(W)}^G$ of the invariant ring is called separating if for any $v,w$ from W that are separated we have that they are separated by S. A separating subset for $𝔽{[V]}^G$ is defined in the same manner. We say that a separating set is minimal if it is minimal w.r.t. inclusion. Obviously, any generating set is also separating. Denote by $\beta \mathrm{\text{sep}}(\mathcal{𝒪}{(W)}^G)$ the minimal integer $\beta \mathrm{\text{sep}}$ such that the set of all invariant polynomial functions of degree less than or equal to $\beta \mathrm{\text{sep}}$ is separating for $\mathcal{𝒪}{(W)}^G$. Minimal separating sets for different actions were constructed in [5, 16, 17, 27, 28, 33, 34, 37].

Separating invariants for $𝔽{[V]}^G$ in case of a finite field $𝔽={𝔽}_q$ with q elements were studied by Kemper, Lopatin and Reimers in [28]. Namely, it was shown that the minimal number of separating invariants for $𝔽{[V]}^G$ is $\gamma (q,\kappa )=\lceil {log}_q(\kappa )\rceil$, where $\kappa$ stands for the number of G-orbits in V, and an explicit construction of a minimal separating set of $\gamma (q,\kappa )$ invariants of degree at most $|G|n(q-1)$ was given. Moreover, a minimal separating set of $\gamma (q,\kappa )$ invariants of degree $\le n(q-1)$ in the non-modular case was constructed as well as in the case when G consists entirely of monomial matrices.

1.3. Matrix invariants

For $n>1$ and $m\ge 1$, the direct sum $M_n^m$ of m copies of the space of $n\times n$ matrices over $𝔽$ is a ${\mathrm{\text{GL}}}_n$-module with respect to the diagonal action by conjugation: $g\cdot \underline{A}=(g{A}_1{g}^{-1},\dots ,g{A}_d{g}^{-1})$ for $g\in {\mathrm{\text{GL}}}_n$ and $\underline{A}=(A_1,\dots ,A_d)$ from $M_n^m$. Given a matrix A, denote by $A_{ij}$ the $(i,j)\mathrm{\text{th}}$ entry of A. Any element of the coordinate ring

of $M_n^m$ can be considered as the polynomial function $x_{ij}(k):M_n^m\to 𝔽$, which sends $(A_1,\dots ,A_m)$ to ${(A_k)}_{ij}$. The algebra of invariant polynomial functions $\mathcal{𝒪}{(M_n^m)}^{{\mathrm{\text{GL}}}_n}$ is defined as in Sec. 1.1.

Sibirskii [38], Procesi [35] and Donkin [18] established that the algebra of invariant polynomial functions $\mathcal{𝒪}{(M_n^m)}^{{\mathrm{\text{GL}}}_n}$ is generated by ${\sigma }_t(X_{k_1}\cdots X_{k_r})$ in case $𝔽$ is infinite, where $X_k$ stands for the $n\times n$ generic matrix ${(x_{ij}(k))}_{1\le i,j\le n}$ and ${\sigma }_t$ stands for the $t\mathrm{\text{th}}$ coefficient of the characteristic polynomial, i.e., $det(\lambda E-A)={\sum }_{t=0}^n{(-1)}^t{\lambda }^{n-t}{\sigma }_t(A)$ for any $n\times n$ matrix A over a commutative ring. In particular, ${\sigma }_0(A)=1$, ${\sigma }_1(A)=\mathrm{\text{tr}}(A)$ and ${\sigma }_n(A)=det(A)$.

For a monomial $c\in 𝔽[M_n^m]$ denote by $degc\in \mathbb{N}$ its degree and by $\mathrm{\text{mdeg}}c\in {\mathbb{N}}^m$ its multidegree, where $\mathbb{N}$ stands for the set of non-negative integers. Namely, $\mathrm{\text{mdeg}}c=(t_1,\dots ,t_d)$, where $t_k$ is the total degree of the monomial c in $x_{ij}(k)$, $1\le i,j\le n$, and $degc=t_1+\cdots +t_m$.

In case of an infinite field of arbitrary characteristic the following minimal separating set for $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{GL}}}_2}$ was given by Kaygorodov, Lopatin and Popov [27]:

Note that set (2) generates the algebra $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{GL}}}_2}$ if and only if the characteristic of $𝔽$ is different from two or $m\le 3$ (see [11, 36]). A minimal generating set for $\mathcal{𝒪}{(M_3^m)}^{{\mathrm{\text{GL}}}_3}$ was given by Lopatin in [29, 30, 31] in the case of an arbitrary infinite field. Over a field of characteristic zero a minimal generating set for $\mathcal{𝒪}{(M_n^2)}^{{\mathrm{\text{GL}}}_n}$ was established by Drensky and Sadikova [19] in case $n=4$ (see also [40]) and by Đoković [8] in case $n=5$ (see also [9]). Some upper bounds on degrees of generating and separating invarints for $\mathcal{𝒪}{(M_n^m)}^{{\mathrm{\text{GL}}}_n}$ were given by Derksen and Makam in [13, 14]. A minimal separating set for the algebra of matrix semi-invariants $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{SL}}}_2\times {\mathrm{\text{SL}}}_2}$ was explicitly described by Domokos [16] over an arbitrary algebraically closed field. Note that a minimal generating set for $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{SL}}}_2\times {\mathrm{\text{SL}}}_2}$ was given by Lopatin [32] over an arbitrary infinite field (see also [17]). Elmer [20, 21] obtained lower bounds on the least possible number of elements for separating sets for $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{GL}}}_2}$ and $\mathcal{𝒪}{(M_2^m)}^{{\mathrm{\text{SL}}}_2\times {\mathrm{\text{SL}}}_2}$ in case $𝔽=ℂ$. More results on separating invariants of matrices can be found in [39].

1.4. Nilpotent matrices

Denote by ${\mathcal{𝒩}}_n$ the set of all $n\times n$ nilpotent matrices, i.e., $A\in M_n$ belongs to ${\mathcal{𝒩}}_n$ if and only if ${\sigma }_1(A)={\sigma }_2(A)=\cdots ={\sigma }_n(A)=0$, or equivalently, $A^n=0$.

Over a finite field, the number of ${\mathrm{\text{GL}}}_n$-orbits on ${\mathcal{𝒩}}_n^m$ was studied by Hua [24]. The variety of pairs of commuting nilpotent matrices has extensively been studied over the past 40 years (see [1, 2, 3, 4, 25, 26] for more details).

The algebra of polynomial functions $\mathcal{𝒪}({\mathcal{𝒩}}_n^m)$ of ${\mathcal{𝒩}}_n^m\subset M_n^m$ is generated by $y_{ij}(k)$ for $1\le i,j\le n$ and $1\le k\le m$, where $y_{ij}(k):{\mathcal{𝒩}}_n^m\to 𝔽$ sends $(A_1,\dots ,A_m)$ to ${(A_k)}_{ij}$. As in Sec. 1.1 we have $\mathcal{𝒪}({\mathcal{𝒩}}_n^m)=\mathcal{𝒪}(M_n^m)∕I({\mathcal{𝒩}}_n^m)$. The algebra $\mathcal{𝒪}{({\mathcal{𝒩}}_n^m)}^{{\mathrm{\text{GL}}}_n}$ of invariant polynomial functions on nilpotent matrices is defined in the same way as $\mathcal{𝒪}{(M_n^m)}^{{\mathrm{\text{GL}}}_n}$. Note that ${\sigma }_t(Y_{k_1}\cdots Y_{k_r})$ lies in $\mathcal{𝒪}{({\mathcal{𝒩}}_n^m)}^{{\mathrm{\text{GL}}}_n}$, where $Y_k={(y_{ij}(k))}_{1\le i,j\le n}$ stands for the generic nilpotent $n\times n$ matrix. Obviously, ${\sigma }_t(Y_k^s)=0$ for all $1\le t\le n$, $s>0$ and $Y_k^n=0$. It is easy to see that in case of an algebraically closed field of zero characteristic the algebra $\mathcal{𝒪}{({\mathcal{𝒩}}_n^m)}^{{\mathrm{\text{GL}}}_n}$ is generated by $\mathrm{\text{tr}}(Y_{i_1}\cdots Y_{i_r})$ (for example, see [5]), but in general case generators for $\mathcal{𝒪}{({\mathcal{𝒩}}_n^m)}^{{\mathrm{\text{GL}}}_n}$ are not known for $n,m\ge 2$. Moreover, over a finite field $𝔽$ generators for algebras $\mathcal{𝒪}{(M_n^m)}^{{\mathrm{\text{GL}}}_n}$ and $𝔽{[M_n^m]}^{{\mathrm{\text{GL}}}_n}$ are not known as well.

Working over an algebraically closed field of zero characteristic Cavalcante and Lopatin in [5] showed that the set

is a minimal generating set and a minimal separating set for the algebra $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ for $m>0$. Minimal generating sets and minimal separating sets for the algebras $\mathcal{𝒪}{({\mathcal{𝒩}}_3^2)}^{{\mathrm{\text{GL}}}_3}$ and $\mathcal{𝒪}{({\mathcal{𝒩}}_3^3)}^{{\mathrm{\text{GL}}}_3}$ were also obtained in [5] in case $p=0$.

1.5. Results

For $𝔽={𝔽}_q$ in Sec. 2 we extend the results from [28] given in Sec. 1.2 to the algebra $\mathcal{𝒪}{(W)}^G$ of G-invariant polynomial functions on a G-invariant subset $W\subset V$ (see Theorem 2.1). Note that a separating set for $\mathcal{𝒪}{(W)}^G$ separates all G-orbits on W in case $𝔽$ is finite (for example, it follows from the proof of Theorem 2.1).

In Sec. 3, working over an arbitrary field, we explicitly describe a minimal set of representatives of all ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^m$ in Theorem 3.4. As a consequence, in case of $𝔽={𝔽}_q$ we explicitly calculate the number of orbits in Corollary 3.5 as well as the least possible number of elements for a separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^2)}^{{\mathrm{\text{GL}}}_2}$. We formulate Conjecture 3.7 about the number of ${\mathrm{\text{GL}}}_n$-orbits on ${\mathcal{𝒩}}_n^m$.

To formulate results about separating sets, introduce some notations. For an arbitrary field $𝔽$ denote

Now assume that $𝔽$ is finite. For all $1\le i,j\le n$ and $\alpha \in {𝔽}^{\times }$ define $\zeta (Y_i)$ and ${\eta }_{\alpha }(Y_i,Y_j)$ from $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ as follows:

where $A,B\in {\mathcal{𝒩}}_2$. The presentations of $\zeta (Y_i)$ and ${\eta }_{\alpha }(Y_i,Y_j)$ as polynomials in $\{y_{ij}(k)\}$ are explicitly given in Sec. 4. In Theorem 4.5, we establish that the set

$$H_{2,m}^{(2)}=S_{2,m}^{(2)}\bigsqcup \{\zeta (Y_i),1\le i\le m;{\eta }_{\alpha }(Y_i,Y_j),1\le i<j\le m,\alpha \in 𝔽\setminus \{0,1\}\}$$

is a minimal separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ in case $\mathrm{\text{char}}𝔽=2$ and the set

$$H_{2,m}=S_{2,m}\bigsqcup \{\zeta (Y_i),1\le i\le m;{\eta }_{\alpha }(Y_i,Y_j),1\le i<j\le m,\alpha \in 𝔽\setminus \{0,1\}\}$$

is a minimal separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ in case $\mathrm{\text{char}}𝔽>2$.

In Theorem 5.4, we describe a minimal separating set in case of an infinite field. Note that over an infinite field the functions $\zeta (Y_i)$ and ${\eta }_{\alpha }(Y_i,Y_j)$ do not belong to $\mathcal{𝒪}({\mathcal{𝒩}}_2^m)$. Some corollaries are given in Sec. 6.

1.6. Notations

If for $\underline{A},\underline{B}\in M_n^m$ there exists $g\in {\mathrm{\text{GL}}}_n$ such that $g\cdot \underline{A}=\underline{B}$, then we write $\underline{A}\sim \underline{B}$ and say that $\underline{A}$, $\underline{B}$ are similar. We say that $\underline{A}$ has no zeros if $A_i$ is non-zero for all i. Denote by $\mathrm{\text{wz}}(\underline{A})$ the result of elimination of all zero matrices from $\underline{A}$. As an example, for $\underline{A}=(0,B,0,C,D)$ with non-zero $B,C,D\in M_n$, we have $\mathrm{\text{wz}}(\underline{A})=(B,C,D)$. For a permutation $\sigma \in {\mathcal{𝒮}}_m$, we write ${\underline{A}}_{\sigma }$ for $(A_{\sigma (1)},\dots ,A_{\sigma (m)})$.

Denote by E the identity matrix and by $E_{ij}$ the matrix such that the $(i,j)\mathrm{\text{th}}$ entry is equal to one and the rest of entries are zeros. Denote by ${𝔽}^{\times }$ the set of all non-zero elements of $𝔽$. For short, we write $\underline{0}$ for $(0,\dots ,0)\in M_n^m$. Given $A\in M_n$ and $r\ge 0$, we write $A^{(r)}$ for $(\underset{r}{\underbrace{A,\dots ,A}})\in M_n^r$.

2. Minimal Separating Invariants for $\mathcal{𝒪}{(W)}^G$ over a Finite Field

In this section, we assume that $𝔽={𝔽}_q$, $W\subset V$ is a G-invariant subset of V, and $dimV=n$. Denote $\gamma =\gamma (q,\kappa )=\lceil {log}_q(\kappa )\rceil$, where $\kappa$ stands for the number of G-orbits on W. For any vector $w=(w_1,\dots ,w_n)\in W$ consider the following polynomial function from $\mathcal{𝒪}(W)$:

Note that $f_w(w)={(-1)}^n{({\prod }_{\alpha \in {𝔽}^{\ast }}\alpha )}^n=1$ and for each $v\in W$ we have

$$f_w(v)=\left\{\begin{array}{cc}1,\hfill & v=w,\hfill \\ 0,\hfill & \mathrm{\text{otherwise.}}\hfill \end{array}\right.$$

We present W as a union of G-orbits: $W=U_1\bigsqcup \cdots \bigsqcup U_{\kappa }$. For every $1\le j\le \kappa$ define

$$f_j=\sum _{u\in U_j}{f}_u.$$(5)

Then for each $v\in W$, we have

$$f_j(v)=\left\{\begin{array}{cc}1\hfill & v\in U_j,\hfill \\ 0\hfill & \mathrm{\text{otherwise}}.\hfill \end{array}\right.$$(6)

Therefore, $f_1,\dots ,f_{\kappa }$ belong to $\mathcal{𝒪}{(W)}^G$ and they form a separating set for $\mathcal{𝒪}{(W)}^G$. By the definition of $\gamma$ we have that $q^{\gamma -1}<\kappa \le q^{\gamma }$. Thus, there exist $\kappa$ pairwise different vectors in ${𝔽}^{\gamma }$, which we may put together in a matrix $({\alpha }_{ij})$ over $𝔽$ of size $(\gamma \times \kappa )$.

Theorem 2.1.

(1)	Every separating set for $\mathcal{𝒪}{(W)}^G$ contains at least $\gamma (q,\kappa )$ elements.
(2)	Let $({\alpha }_{ij})\in {𝔽}^{\gamma \times \kappa }$ be a matrix whose columns are pairwise different and define $$h_i={\alpha }_{i1}{f}_1+\cdots +{\alpha }_{i\kappa }{f}_{\kappa }(1\le i\le \gamma ).$$ Then $\{h_1,\dots ,h_{\gamma }\}$ is a minimal separating set for $\mathcal{𝒪}{(W)}^G$ consisting of (non-homogeneous) elements of degree less than or equal to $n(q-1)$.

Proof. Consider some representatives of G-orbits on W: $u_1\in U_1,\dots ,u_{\kappa }\in U_{\kappa }$.

(1) Assume that $\{l_1,\dots ,l_r\}$ is a separating set for $\mathcal{𝒪}{(W)}^G$. Then the column vectors

are pairwise different. Hence, $\kappa \le q^r$ and $\gamma \le r$ follows from the definition of $\gamma$.

(2) Applying formula (6), we obtain that

By the conditions of the theorem, these column vectors are pairwise different, hence $\{h_1,\dots ,h_{\gamma }\}$ is a separating set. The minimality follows from part 1. □

3. Classification of ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^m$

In this section we assume that $𝔽$ is an arbitrary field.

Lemma 3.1. If $\underline{A}\in {\mathcal{𝒩}}_2^2$ has no zeros, then $\underline{A}\sim (E_{12},\alpha E_{12})$ or $\underline{A}\sim (E_{12},\alpha E_{21})$ for some non-zero $\alpha \in 𝔽$.

Proof. For each field $𝔽$ we have that $A_1\sim E_{12}$. Therefore, without loss of generality, we can assume that $A_1=E_{12}$. Denote $A_2=\left(\begin{array}{cc}\hfill a_1\hfill & \hfill a_2\hfill \\ \hfill a_3\hfill & \hfill -a_1\hfill \end{array}\right)$ for some $a_1,a_2,a_3\in 𝔽$ and assume that $g=\left(\begin{array}{cc}\hfill g_1\hfill & \hfill g_2\hfill \\ \hfill 0\hfill & \hfill g_1\hfill \end{array}\right)$ lies in ${\mathrm{\text{GL}}}_2$. Then we have that $g\cdot A_1=A_1$ and

for $h_1=a_1+\frac{g_2}{g_1}{a}_3$ and $h_2=(a_2{g}_1^2-2a_1{g}_1{g}_2-a_3{g}_2^2)∕g_1^2$.

Assume that $a_3\ne 0$. Then we take $g_1=1$ and $g_2=-\frac{a_1}{a_3}$ to obtain that $h_1=0$. Since $g\cdot A_2$ is nilpotent, we obtain $h_2=0$ and $g\cdot A_2=a_3{E}_{21}$.

Assume that $a_3=0$. Since $A_2$ is nilpotent, we obtain $A_2=a_2{E}_{12}$. □

For $\beta ,\gamma ,\delta$ of $𝔽$ denote

Remark 3.2. Assume that $\underline{A},{\underline{A}}^{\prime }\in {\mathcal{𝒩}}_2^3$ satisfy $A_1=A_1^{\prime }=E_{12}$, $A_2=\alpha E_{21}$, and $A_2^{\prime }={\alpha }^{\prime }{E}_{21}$ for some $\alpha ,{\alpha }^{\prime }\in {𝔽}^{\times }$. Assume that $t_{ij}:=\mathrm{\text{tr}}(A_i{A}_j)-\mathrm{\text{tr}}(A_i^{\prime }{A}_j^{\prime })=0$ for all $1\le i<j\le 3$ and $t_{123}:=\mathrm{\text{tr}}(A_1{A}_2{A}_3)-\mathrm{\text{tr}}(A_1^{\prime }{A}_2^{\prime }{A}_3^{\prime })=0$. Then $\underline{A}={\underline{A}}^{\prime }$.

Proof. Denote $A_3=D(\beta ,\gamma ,\delta )$ and $A_3^{\prime }=D({\beta }^{\prime },{\gamma }^{\prime },{\delta }^{\prime })$ for some $\beta ,\gamma ,\delta ,{\beta }^{\prime },{\gamma }^{\prime },{\delta }^{\prime }$ from $𝔽$. Since $t_{12}=0$, we have $\alpha ={\alpha }^{\prime }$. Consequently applying the equalities $t_{13}=0$, $t_{23}=0$, $t_{123}=0$, we obtain that $\delta ={\delta }^{\prime }$, $\alpha \gamma =\alpha {\gamma }^{\prime }$, $\alpha \beta =\alpha {\beta }^{\prime }$. □

Lemma 3.3. If $\underline{A}\in {\mathcal{𝒩}}_2^m$ has no zeros, then $\underline{A}$ is similar to one and only one of the following elements:

(a)	$(E_{12},{\alpha }_2{E}_{12},\dots ,{\alpha }_m{E}_{12})$, where ${\alpha }_2,\dots ,{\alpha }_m\in {𝔽}^{\times }$;
(b)	$(E_{12},{\alpha }_2{E}_{12},\dots ,{\alpha }_r{E}_{12},{\alpha }_{r+1}{E}_{21},D_1,\dots ,D_s)$, where $1\le r\le m-1$, $s=m-r-1$, ${\alpha }_2,\dots ,{\alpha }_{r+1}\in {𝔽}^{\times }$, $D_i\in {\mathcal{𝒩}}_2$ is non-zero for every i.

Proof. In case $m=1$, we have $A_1\sim E_{12}$ and the required is proven.

Assume $m\ge 2$. By Lemma 3.1 either $(A_1,A_2)\sim (E_{12},{\alpha }_2{E}_{12})$ or $(A_1,A_2)\sim (E_{12},{\alpha }_2{E}_{21})$ for some ${\alpha }_2\in {𝔽}^{\times }$. In the first case we have $A_1=A_2∕{\alpha }_2$ and applying Lemma 3.1 to the pair $(A_1,A_3)$ we can see that either $(A_1,A_2,A_3)\sim (E_{12},{\alpha }_2{E}_{12},{\alpha }_3{E}_{12})$ or $(A_1,A_2,A_3)\sim (E_{12},{\alpha }_2{E}_{12},{\alpha }_3{E}_{21})$ for some ${\alpha }_3\in {𝔽}^{\times }$. Repeating this procedure, we finally obtain that either case (a) or (b) holds.

To prove uniqueness, consider some $\underline{A},{\underline{A}}^{\prime }\in {\mathcal{𝒩}}_2^m$ of type (a) or (b) and satisfying the condition $\underline{A}\sim {\underline{A}}^{\prime }$. Note that $\underline{A}$ has type (a) if and only if for every $2\le i\le m$ exists ${\alpha }_i\in {𝔽}^{\times }$ such that $A_1=A_i∕{\alpha }_i$. Since this property of $\underline{A}$ is not changed by the action of ${\mathrm{\text{GL}}}_2$, then we have one of the following two cases.

(1) Both $\underline{A}$ and ${\underline{A}}^{\prime }$ have type (a), i.e.,

where ${\alpha }_2,\dots ,{\alpha }_m,{\alpha }_2^{\prime },\dots ,{\alpha }_m^{\prime }\in {𝔽}^{\times }$. Consider $g\in {\mathrm{\text{GL}}}_2$ such that $g\cdot \underline{A}={\underline{A}}^{\prime }$. Since $g\cdot A_i=A_i^{\prime }$ for all i, we obtain that $g\cdot E_{12}=E_{12}$ and ${\alpha }_i={\alpha }_i^{\prime }$ for $2\le i\le m$. Therefore, $\underline{A}={\underline{A}}^{\prime }$.

(2) $\underline{A}$ and ${\underline{A}}^{\prime }$ have both type (b), i.e.,

where $r,r^{\prime }\ge 1$, $s,s^{\prime }\ge 0$, ${\alpha }_2,\dots ,{\alpha }_{r+1},{\alpha }_2^{\prime },\dots ,{\alpha }_{r^{\prime }+1}^{\prime }\in {𝔽}^{\times }$, $D_i,D_i^{\prime }\in {\mathcal{𝒩}}_2$ is non-zero for every i. By the definition of similarity we have that $t_{ij}:=\mathrm{\text{tr}}(A_i{A}_j)-\mathrm{\text{tr}}(A_i^{\prime }{A}_j^{\prime })=0$ and $t_{ijk}:=\mathrm{\text{tr}}(A_i{A}_j{A}_k)-\mathrm{\text{tr}}(A_i^{\prime }{A}_j^{\prime }{A}_k^{\prime })=0$ for all $1\le i,j,k\le m$. Since we obtain that $r=r^{\prime }$ and $s=s^{\prime }$. Consider $1\le i\le s$. Applying Remark 3.2 to the pair of triples $(A_1,A_{r+1},A_{r+i+1})$ and $(A_1^{\prime },A_{r+1}^{\prime },A_{r+i+1}^{\prime })$, we obtain that ${\alpha }_{r+1}={\alpha }_{r+1}^{\prime }$ and $A_{r+i+1}=A_{r+i+1}^{\prime }$. In case $r\ge 2$ we use $t_{j,r+1}=0$ to obtain that ${\alpha }_j={\alpha }_j^{\prime }$ for all $2\le j\le r$. Therefore, $\underline{A}={\underline{A}}^{\prime }$. □

Lemma 3.3 implies the following description of ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^m$.

Theorem 3.4. Each ${\mathrm{\text{GL}}}_2$-orbit on ${\mathcal{𝒩}}_2^m$ contains one and only one element of the following types:

(0)	$(0,\dots ,0)$;
(1)	$({\alpha }_1{E}_{12},\dots ,{\alpha }_m{E}_{12})$, where • at least one element from the list ${\alpha }_1,\dots ,{\alpha }_m\in 𝔽$ is non-zero, • ${\alpha }_v=1$ for $v=min\{1\le i\le m|{\alpha }_i\ne 0\}$;
(2)	$({\alpha }_1{E}_{12},\dots ,{\alpha }_r{E}_{12},{\alpha }_{r+1}{E}_{21},D_1,\dots ,D_s)$, where • $1\le r\le m-1$, $s=m-r-1$, • at least one element from the set $\{{\alpha }_1,\dots ,{\alpha }_r\}$ is non-zero and ${\alpha }_{r+1}\in {𝔽}^{\times }$, • ${\alpha }_v=1$ for $v=min\{1\le i\le r|{\alpha }_i\ne 0\}$, • $D_i\in {\mathcal{𝒩}}_2$ for every $1\le i\le s$.

For every integer k denote

Corollary 3.5. Assume $𝔽={𝔽}_q$ and $m\ge 1$. Then

(a)	the number of ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^m$ is equal to $$\kappa =1+\frac{(q^m-1)(q^{m-1}+q)}{q^2-1};$$
(b)	$\kappa$ has the following presentation as a polynomial in q with non-negative integer coefficients: $$\kappa =\kappa (q)=\left\{\begin{array}{cc}1+S_{m-1}(q)+(q^{m-1}-1)S_{\frac{m-2}{2}}(q^2)\hfill & \mathit{if}m\mathit{is even},\hfill \\ 1+S_{m-1}(q)+(q^m-1)S_{\frac{m-3}{2}}(q^2)\hfill & \mathit{if}m\mathit{is odd},\hfill \end{array}\right.$$
(c)	the least possible number of elements for a separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ is $$\gamma =\left\{\begin{array}{cc}1\hfill & \mathit{if}m=1,\hfill \\ 3\hfill & \mathit{if}m=q=2,\hfill \\ 2m-2\hfill & \mathit{otherwise. }\hfill \end{array}\right.$$

Proof. (a) Denote by ${\kappa }_0(m)$, ${\kappa }_1(m)$, ${\kappa }_2(m)$, respectively, the number of ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^m$ of type (0), (1), (2), respectively (see Theorem 3.4). Then ${\kappa }_0(m)=1$, ${\kappa }_1(m)=S_{m-1}(q)$, and

since we apply the equality $|{\mathcal{𝒩}}_n|=q^{n(n-1)}$, which was proven by Fine and Herstein [22]. Hence,

$${\kappa }_2(m)=q^{2m-2}\left(\sum _{r=0}^{m-1}{q}^{-r}-\sum _{r=0}^{m-1}{q}^{-2r}\right)=q^{2m-2}\left(S_{m-1}(q^{-1})-S_{m-1}(q^{-2})\right),$$

and we obtain

$${\kappa }_2(m)=\frac{(q^m-1)(q^{m-1}-1)}{q^2-1}.$$

Therefore,

$$\kappa ={\kappa }_0(m)+{\kappa }_1(m)+{\kappa }_2(m)=1+S_{m-1}(q)+\frac{(q^m-1)(q^{m-1}-1)}{q^2-1},$$(7)

and the claim of part (a) easily follows.

(b) If $m=1$, then $\kappa =2$ and part (b) holds. Assume $m\ge 2$. Considering the case of even m and the case of odd m, applying formula (7), we prove the claim of part (b).

(c) By Theorem 2.1, the least possible number of elements for a separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ is $\gamma =\lceil {log}_q(\kappa )\rceil$.

If $m=1$, then $\kappa =2$ and $\gamma =1$.

If $m=q=2$, then $\kappa =5$ and $\gamma =3$.

Assume $m\ge 2$ and $m,q$ are not simultaneously equal to 2. By part (a), the claim that $\gamma =2m-2$ is equivalent to inequalities

The left inequality follows from:

$$q^{2m-3}<\frac{(q^m-1)(q^{m-1}+q)}{q^2-1}.$$

The right inequality is equivalent to

$$(q^{2m}-q^{2m-1}-q^{2m-2}-q^{m+1})+(q^{m-1}-q^2)+q+1\ge 0.$$(8)

In case $m=2$ and $q\ge 3$, inequality (8) is equivalent to $q^2(q^2-2q-2)+2q+1\ge 0$, which holds since $q^2-2q-2\ge 0$ for $q\ge 3$.

In case $m\ge 3$ we have $q^{2m}=q^{2m-2}{q}^2\ge q^{2m-2}q+q^{2m-2}+q^{2m-2}\ge q^{2m-1}+q^{2m-2}+q^{m+1}$, since $q^2\ge q+2$ for $q\ge 2$; inequality (8) follows. □

Corollary 3.5 implies the following remark:

Remark 3.6. Assume $𝔽={𝔽}_q$. Then

•	for $m=1$ we have $\kappa =2$;
•	for $m=2$ we have $\kappa =2q+1$;
•	for $m=3$ we have $\kappa =q^3+q^2+q+1$;
•	for $m=4$ we have $\kappa =q^5+2q^3+q+1$;
•	for $m=5$ we have $\kappa =q^7+q^5+q^4+q^3+q+1$.

Corollary 3.5 implies that the following conjecture holds for $n=2$ and $m\ge 1$.

Conjecture 3.7. Assume $𝔽={𝔽}_q$. If we fix $n\ge 2$ and $m\ge 1$, then the number of ${\mathrm{\text{GL}}}_n$-orbits on ${\mathcal{𝒩}}_n^m$ is a polynomial in q with integer coefficients.

It was proven by Hua [24] that the number of ${\mathrm{\text{GL}}}_n$-orbits on ${\mathcal{𝒩}}_n^m$ is a polynomial in q with rational coefficients. Hua also conjectured that the number of absolutely indecomposable ${\mathrm{\text{GL}}}_n$-orbits on ${\mathcal{𝒩}}_n^m$ is a polynomial in q with non-negative integer coefficients (see [24, Conjecture 4.1]). This conjecture was shown to be true for $m=2$ and $1\le n\le 6$.

4. Separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$

If $𝔽={𝔽}_q$ and $A=\left(\begin{array}{cc}\hfill a_1\hfill & \hfill a_2\hfill \\ \hfill a_3\hfill & \hfill a_4\hfill \end{array}\right)$ lies in ${\mathcal{𝒩}}_2$, then we have that $\zeta (A)=a_1^{q-1}-a_2^{q-1}-a_3^{q-1}+1$, since $a_1^2=-a_2{a}_3$. In particular,

Similarly, we can see that

$${\eta }_{\alpha }(Y_i,Y_j)=\prod _{u,v\in \{1,2\}}({(\alpha y_{uv}(i)-y_{uv}(j))}^{q-1}-1).$$(10)

Remark 4.1. Let H be one of the following sets: $S_{2,m}$, $S_{2,m}^{(2)}$, $H_{2,m}$, $H_{2,m}^{(2)}$. Assume that $\underline{A},\underline{B}\in {\mathcal{𝒩}}_2^m$ are not separated by H. Then

(a)	for any $\sigma \in {\mathcal{𝒮}}_m$ we have that ${\underline{A}}_{\sigma },{\underline{B}}_{\sigma }$ are not separated by H;
(b)	for any ${\underline{A}}^{\prime },{\underline{B}}^{\prime }\in {\mathcal{𝒩}}_n^m$ with $\underline{A}\sim {\underline{A}}^{\prime }$ and $\underline{B}\sim {\underline{B}}^{\prime }$ we have that ${\underline{A}}^{\prime },{\underline{B}}^{\prime }$ are not separated by H.

Proof. Consider some $A_1,A_2,A_3$ from ${\mathcal{𝒩}}_2$. It is easy to see that ${\eta }_{\alpha }(A_1,A_2)={\eta }_{{\alpha }^{-1}}(A_2,A_1)$ for all $\alpha \in {𝔽}^{\times }$. Moreover, the Cayley–Hamilton theorem implies that

Thus, part (a) is proven. Part (b) is trivial. □

Lemma 4.2. Assume that $\underline{A},\underline{B}\in {\mathcal{𝒩}}_2^m$ are not separated by $S_{2,m}$. Then there exists a permutation $\sigma \in {\mathcal{𝒮}}_m$ such that one of the next cases holds:

(a)

$$\begin{array}{ccc}\hfill {\underline{A}}_{\sigma }& \hfill \sim \hfill & (0^{(k+l)},{\alpha }_1{E}_{12},\dots ,{\alpha }_{r+s}{E}_{12}),\hfill \\ \hfill {\underline{B}}_{\sigma }& \hfill \sim \hfill & (0^{(k)},{\beta }_1{E}_{12},\dots ,{\beta }_l{E}_{12},0^{(r)},{\beta }_{l+1}{E}_{12},\dots ,{\beta }_{l+s}{E}_{12}),\hfill \end{array}$$ where $k,l,r,s\ge 0,k+l+r+s=m$, ${\alpha }_1,\dots ,{\alpha }_{r+s},{\beta }_1,\dots ,{\beta }_{l+s}\in {𝔽}^{\times }$; moreover, ${\alpha }_1=1$ in case $\underline{A}\ne \underline{0}$ and ${\beta }_1=1$ in case $\underline{B}\ne \underline{0}$;

(b)

${\underline{A}}_{\sigma }\sim (0^{(k)},E_{12},{\alpha }_2{E}_{12},\dots ,{\alpha }_r{E}_{12},{\alpha }_{r+1}{E}_{21},D_1,\dots ,D_s)$ and $\underline{B}\sim \underline{A}$, where $r\ge 1$, $k,s\ge 0$, $k+r+s+1=m$, ${\alpha }_2,\dots ,{\alpha }_{r+1}\in {𝔽}^{\times }$, and $D_i\in {\mathcal{𝒩}}_2$ is non-zero for every i.

Proof. We use Remark 4.1 without reference to it. Applying Lemma 3.3 to $\underline{A}$ together with ${\mathcal{𝒮}}_m$-action on $\underline{A}$ we can see that one of the next two cases holds.

(1) ${\underline{A}}_{\sigma }\sim (0^{(k)},{\alpha }_1{E}_{12},\dots ,{\alpha }_r{E}_{12})$, where $k,r\ge 0$, $k+r=m$, and ${\alpha }_1,\dots ,{\alpha }_r\in {𝔽}^{\times }$ with ${\alpha }_1=1$ in case $\underline{A}\ne \underline{0}$. Applying Lemma 3.3 to ${\underline{B}}_{\sigma }$ together with ${\mathcal{𝒮}}_m$-action on $({\underline{A}}_{\sigma },{\underline{B}}_{\sigma })$ we can see that one of the next two cases holds:

(1a)

there is $\tau \in {\mathcal{𝒮}}_m$ such that $({\underline{A}}_{\tau },{\underline{B}}_{\tau })$ satisfies case (a) maybe without condition that ${\beta }_1=1$ in case $\underline{B}\ne \underline{0}$. In the latter case we act by $\left(\begin{array}{cc}\hfill {\beta }_1^{-1}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right)\in {\mathrm{\text{GL}}}_2$ on ${\underline{B}}_{\tau }$ to obtain case (a).

(1b)

${\underline{A}}_{\sigma }$ is the same as above and ${\underline{B}}_{\sigma }\sim \underline{C}$, where for some $i<j$ we have $C_i=E_{12}$ and $C_j=\alpha E_{21}$ for $\alpha \in {𝔽}^{\times }$. In this case, we have $\mathrm{\text{tr}}(A_{\sigma (i)}{A}_{\sigma (j)})=\mathrm{\text{tr}}(C_i{C}_j)$; hence, $\alpha =0$, a contradiction.

(2) ${\underline{A}}_{\sigma }\sim (0^{(k)},{\alpha }_1{E}_{12},\dots ,{\alpha }_r{E}_{12},{\alpha }_{r+1}{E}_{21},D_1,\dots ,D_s)$, where $r\ge 1$, $k,s\ge 0$, ${\alpha }_1=1$, ${\alpha }_2,\dots ,{\alpha }_{r+1}\in {𝔽}^{\times }$, and $D_i\in {\mathcal{𝒩}}_2$ is non-zero for every i. Applying Lemma 3.3 to ${\underline{B}}_{\sigma }$ we can see that one of the next two cases holds:

(2a)

$wz({\underline{B}}_{\sigma })\sim ({\beta }_1{E}_{12},\dots ,{\beta }_l{E}_{12})$ for some $l\ge 0$ and ${\beta }_1,\dots ,{\beta }_l\in {𝔽}^{\times }$, where ${\beta }_1=1$ in case $\underline{B}\ne \underline{0}$. Then $B_i{B}_j=0$ for all $1\le i,j\le m$. On the other hand, $\mathrm{\text{tr}}(A_{\sigma (k+r)}{A}_{\sigma (k+r+1)})={\alpha }_r{\alpha }_{r+1}$ is non-zero; a contradiction.

(2b)

$wz({\underline{B}}_{\sigma })\sim ({\beta }_1{E}_{12},\dots ,{\beta }_{r^{\prime }}{E}_{12},{\beta }_{r^{\prime }+1}{E}_{21},D_1^{\prime },\dots ,D_{s^{\prime }}^{\prime })$, where $r^{\prime }\ge 1$, $s^{\prime }\ge 0$, ${\beta }_1=1$, ${\beta }_2,\dots ,{\beta }_{r^{\prime }+1}\in {𝔽}^{\times }$, $D_i^{\prime }\in {\mathcal{𝒩}}_2$ is non-zero for every $1\le i\le s^{\prime }$. Therefore, $$\begin{array}{ccc}\hfill & \hfill {\underline{B}}_{\sigma }\sim (\dots ,{\beta }_{r^{\prime }}{E}_{12},0,\dots ,0,{\beta }_{r^{\prime }+1}{E}_{21},\dots ),\hfill & \hfill \\ \hfill & \hfill îĵ\hfill & \hfill \end{array}$$ where ${\beta }_{r^{\prime }}{E}_{12}$ and ${\beta }_{r^{\prime }+1}{E}_{21}$, respectively, stands in position i and j, respectively, for some $1\le i<j\le m$. Since $$\begin{array}{ccc}\hfill min\{1\le v\le m|\exists 1\le u<v\mathrm{\text{such that}}\mathrm{\text{tr}}(A_{\sigma (u)}{A}_{\sigma (v)})\ne 0\}& \hfill =\hfill & k+r+1,\hfill \\ \hfill min\{1\le v\le m|\exists 1\le u<v\mathrm{\text{such that}}\mathrm{\text{tr}}(B_{\sigma (u)}{B}_{\sigma (v)})\ne 0\}& \hfill =\hfill & j,\hfill \end{array}$$ the conditions of the lemma imply that $j=k+r+1$. For $1\le u\le k+r$, we have $$\mathrm{\text{tr}}(B_{\sigma (u)}{B}_{\sigma (k+r+1)})=\mathrm{\text{tr}}(A_{\sigma (u)}{A}_{\sigma (k+r+1)})=\left\{\begin{array}{cc}{\alpha }_{u-k}{\alpha }_{r+1}\hfill & \mathrm{\text{if}}u>k,\hfill \\ 0\hfill & \mathrm{\text{if}}u\le k.\hfill \end{array}\right.$$(12) Therefore, ${\underline{B}}_{\sigma }\sim (0^{(k)},{\beta }_1{E}_{12},\dots ,{\beta }_r{E}_{12},{\beta }_{r+1}{E}_{21},D_1^{″},\dots ,D_s^{″})$, where some of the matrices $D_1^{″},\dots ,D_s^{″}\in {\mathcal{𝒩}}_2$ can be zero. In particular, $r=r^{\prime }$. Since ${\alpha }_1={\beta }_1=1$, equality (12) implies that ${\alpha }_{r+1}={\beta }_{r+1}$; therefore, ${\alpha }_i={\beta }_i$ for all $2\le i\le r$. Applying Remark 3.2 to the pair of triples $(A_{\sigma (k+r)},A_{\sigma (k+r+1)},A_{\sigma (k+r+1+u)})$ and $(B_{\sigma (k+r)},B_{\sigma (k+r+1)},B_{\sigma (k+r+1+u)})$, where $1\le u\le s$, we obtain that $D_u=D_u^{\prime }$. Therefore, ${\underline{A}}_{\sigma }\sim {\underline{B}}_{\sigma }$ and case (b) of the formulation of this lemma holds.

 □

Remark 4.3. For $\underline{A}\in {\mathcal{𝒩}}_2^2$, we have

Proof. Since ${\eta }_{\alpha }(A_1,A_2)$ and $\mathrm{\text{tr}}(A_1{A}_2)$ are constants on the ${\mathrm{\text{GL}}}_2$-orbits on ${\mathcal{𝒩}}_2^2$, by Lemma 3.1 we can assume that on $\underline{A}$ lies in the following list: $(0,0)$, $(0,E_{12})$, $(E_{12},0)$, $(E_{12},\gamma E_{12})$, $(E_{12},\gamma E_{21})$, where $\gamma \in {𝔽}^{\times }$. The required follows from case by case consideration. □

Lemma 4.4. Assume that $\mathrm{\text{char}}𝔽=2$ and $\underline{A},\underline{B}\in {\mathcal{𝒩}}_2^3$ are not separated by $S_3^{(2)}=\{\mathrm{\text{tr}}(Y_1{Y}_2),\mathrm{\text{tr}}(Y_1{Y}_3),\mathrm{\text{tr}}(Y_2{Y}_3)\}$. Then $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=\mathrm{\text{tr}}(B_1{B}_2{B}_3)$.

Proof. If $A_i=B_j=0$ for some $1\le i,j\le 3$, then $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=\mathrm{\text{tr}}(B_1{B}_2{B}_3)=0$. Therefore, without loss of generality, we can assume that $A_i$ is non-zero for all i.

(1) Assume that $B_j=0$ for some j. By formula (11), without loss of generality, we can assume that $B_1=0$. Applying Lemma 3.3 to $\underline{A}$, we obtain that one of the following cases holds:

where ${\alpha }_2,{\alpha }_3\in {𝔽}^{\times }$ and $D\in {\mathcal{𝒩}}_2$ is non-zero. In cases (a) and (c), we have $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=0$ and the required holds. In case (b), we have ${\alpha }_2=\mathrm{\text{tr}}(A_1{A}_2)=\mathrm{\text{tr}}(B_1{B}_2)=0$; a contradiction.

(2) Assume that $B_j$ is non-zero for all j. Applying Lemma 3.3 to $\underline{A}$, we obtain that one of the above cases (a), (b), (c) holds for $\underline{A}$. Applying Lemma 3.3 to $\underline{B}$, we obtain that one of the following cases holds:

where ${\beta }_2,{\beta }_3\in {𝔽}^{\times }$ and $D^{\prime }\in {\mathcal{𝒩}}_2$ is non-zero. Since $\mathrm{\text{tr}}(A_1{A}_2)=\mathrm{\text{tr}}(B_1{B}_2)$ and $\mathrm{\text{tr}}(A_1{A}_3)=\mathrm{\text{tr}}(B_1{B}_3)$, we have one of the following three cases:

•	Cases (a) and ($\mathrm{\text{a’}}$) hold. Then $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=\mathrm{\text{tr}}(B_1{B}_2{B}_3)=0$.
•	Cases (b) and ($\mathrm{\text{b’}}$) hold. Since $\mathrm{\text{tr}}(A_1{A}_2)=\mathrm{\text{tr}}(B_1{B}_2)$, we obtain that ${\alpha }_2={\beta }_2$. Denote $D=D(a_1,a_2,a_3)$ and $D^{\prime }=D(b_1,b_2,b_3)$. The equalities $\mathrm{\text{tr}}(A_1{A}_3)=\mathrm{\text{tr}}(B_1{B}_3)$ and $\mathrm{\text{tr}}(A_2{A}_3)=\mathrm{\text{tr}}(B_2{B}_3)$, respectively, imply that $a_3=b_3$ and $a_2=b_2$, respectively. It follows from equalities $det(D)=det(D^{\prime })=0$ that $a_1=b_1$, since $\mathrm{\text{char}}𝔽=2$. Therefore, $\underline{A}\sim \underline{B}$ and the required follows.
•	Cases (c) and ($\mathrm{\text{c’}}$) hold. Then $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=\mathrm{\text{tr}}(B_1{B}_2{B}_3)=0$.

 □

Theorem 4.5. Assume that $𝔽={𝔽}_q$ is finite. Then

•	$H_{2,m}^{(2)}$, in case $\mathrm{\text{char}}𝔽=2$;
•	$H_{2,m}$, in case $\mathrm{\text{char}}𝔽>2$;

is a minimal separating set for the algebra of invariant polynomial functions on nilpotent matrices $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ for all $m>0$.

Proof. Denote by H the set $H_{2,m}^{(2)}$ or $H_{2,m}$, respectively, in case $\mathrm{\text{char}}𝔽=2$ or $\mathrm{\text{char}}𝔽>2$, respectively. Assume that $\underline{A},\underline{B}\in {\mathcal{𝒩}}_2^m$ are not separated by H. To prove that H is separating it is enough to show that $\underline{A}\sim \underline{B}$. By Remarks 4.1, 4.3 and Lemma 4.4, we obtain in both cases that $\underline{A},\underline{B}$ are not separated by

Applying Lemma 4.2 together with the fact $A_i=0$ if and only if $B_i=0$ ($1\le i\le m$), we obtain that one of the next two cases holds:

•	${\underline{A}}_{\sigma }\sim (0^{(k)},E_{12},{\alpha }_{k+2}{E}_{12},\dots ,{\alpha }_m{E}_{12})$, ${\underline{B}}_{\sigma }\sim (0^{(k)},E_{12},{\beta }_{k+2}{E}_{12},\dots ,{\beta }_m{E}_{12})$, where $\sigma \in {\mathcal{𝒮}}_m$, $0\le k<m$, ${\alpha }_{k+2},\dots ,{\alpha }_m,{\beta }_{k+2},\dots ,{\beta }_m\in {𝔽}^{\times }$;
•	$\underline{A}\sim \underline{B}$.

Assume that the first case holds. Since for every $k+2\le i\le m$ and every $\alpha \in {𝔽}^{\times }$ we have

then ${\alpha }_i={\beta }_i$. Therefore, ${\underline{A}}_{\sigma }\sim {\underline{B}}_{\sigma }$ and H is separating.

Claims 1–4 (see below) show that H is a minimal separating set for all $m>0$.

Claim 1. The set $H_{2,1}\setminus \{\zeta (Y_1)\}=\varnothing$  is not separating for $\mathcal{𝒪}{({\mathcal{𝒩}}_2)}^{{\mathrm{\text{GL}}}_2}$ .

To prove this claim it is enough to consider $\underline{A}=(0)$ and $\underline{B}=(E_{12})$.

Claim 2. Given $\beta \in 𝔽\setminus \{0,1\}$ , the set $H_{2,2}\setminus \{{\eta }_{\beta }(Y_1,Y_2)\}$  is not separating for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^2)}^{{\mathrm{\text{GL}}}_2}$ .

To prove this claim consider $\underline{A}=(E_{12},E_{12})$ and $\underline{B}=(E_{12},\beta E_{12})$ from ${\mathcal{𝒩}}_2^2$. Then $\mathrm{\text{tr}}(A_1{A}_2)=\mathrm{\text{tr}}(B_1{B}_2)=0$ and ${\eta }_{\alpha }(A_1,A_2)={\eta }_{\alpha }(B_1,B_2)=0$ for all $\alpha \in 𝔽\setminus \{0,1,\beta \}$, but ${\eta }_{\beta }(A_1,A_2)\ne {\eta }_{\beta }(B_1,B_2)$.

Claim 3. The set $H_{2,2}\setminus \{\mathrm{\text{tr}}(Y_1{Y}_2)\}$  is not separating for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^2)}^{{\mathrm{\text{GL}}}_2}$ .

To prove this claim consider $\underline{A}=(E_{12},E_{12})$ and $\underline{B}=(E_{12},E_{21})$ from ${\mathcal{𝒩}}_2^2$. Then ${\eta }_{\alpha }(A_1,A_2)={\eta }_{\alpha }(B_1,B_2)=0$ for all $\alpha \in 𝔽\setminus \{0,1\}$, but $\mathrm{\text{tr}}(A_1{A}_2)\ne \mathrm{\text{tr}}(B_1{B}_2)$.

Claim 4. Let $m=3$  and $\mathrm{\text{char}}𝔽\ne 2$ . Then $H_{2,3}\setminus \{\mathrm{\text{tr}}(Y_1{Y}_2{Y}_3)\}$  is not separating for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^3)}^{{\mathrm{\text{GL}}}_2}$ .

To prove this claim we consider $\underline{A}=(E_{12},E_{21},A_3)$  and $\underline{B}=(E_{12},E_{21},B_3)$  from ${\mathcal{𝒩}}_2^3$ , where

Thus, ${\eta }_{\alpha }(A_i,A_j)={\eta }_{\alpha }(B_i,B_j)=0$ for all $1\le i<j\le 3$ and $\alpha \in 𝔽\setminus \{0,1\}$. Moreover, $\mathrm{\text{tr}}(A_1{A}_2)=\mathrm{\text{tr}}(B_1{B}_2)=1$, $\mathrm{\text{tr}}(A_1{A}_3)=\mathrm{\text{tr}}(B_1{B}_3)=-1$, $\mathrm{\text{tr}}(A_2{A}_3)=\mathrm{\text{tr}}(B_2{B}_3)=1$. On the other hand, $\mathrm{\text{tr}}(A_1{A}_2{A}_3)=1$ and $\mathrm{\text{tr}}(B_1{B}_2{B}_3)=-1$ are different. □

5. The Case of Infinite Field

In this section we assume that $𝔽$ is infinite. We say that a G-invariant subset $W\subset V$ is a G-invariant cone if the conditions $w\in W$ and $\alpha \in {𝔽}^{\times }$ imply that $\alpha w\in W$.

Remark 5.1. If $W\subset V$ is a G-invariant cone, then the algebra $\mathcal{𝒪}{(W)}^G$ has $\mathbb{N}$-grading by degrees.

Note that ${\mathcal{𝒩}}_n^m$ is a multi-cone, i.e., if $\underline{A}\in {\mathcal{𝒩}}_n^m$ and ${\alpha }_1,\dots ,{\alpha }_m\in {𝔽}^{\times }$, then $({\alpha }_1{A}_1,\dots ,{\alpha }_m{A}_m)\in {\mathcal{𝒩}}_n^m$. Thus, the following extension of Remark 5.1 to the case of multidegree holds.

Remark 5.2. The algebra $\mathcal{𝒪}{({\mathcal{𝒩}}_n^m)}^{{\mathrm{\text{GL}}}_n}$ has ${\mathbb{N}}^m$-grading by multidegrees, where the multidegree of $f\in \mathcal{𝒪}({\mathcal{𝒩}}_n^m)\simeq 𝔽[M_n^m]∕I({\mathcal{𝒩}}_n^m)$ is defined as the minimal multidegree (with respect to some fixed lexicographical order) of a polynomial $h\in 𝔽[M_n^m]$ with $f=h+I({\mathcal{𝒩}}_n^m)$.

Lemma 5.3. If $f\in \mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ is ${\mathbb{N}}^m$-homogeneous and $f\notin 𝔽$, then $f({\alpha }_1{E}_{12},\dots ,{\alpha }_m{E}_{12})=0$ for all ${\alpha }_1,\dots ,{\alpha }_m\in 𝔽$.

Proof. Denote the multidegree of f by $\underline{\delta }=({\delta }_1,\dots ,{\delta }_m)$, where ${\delta }_{!}+\cdots +{\delta }_m>0$. Then

for some $\beta ,{\beta }_h\in 𝔽$, where $\mathrm{\Omega }$ is the set of all monomials in $\{y_{ij}(k)|i,j\in \{1,2\},1\le k\le m\}$ different from $y_{12}{(1)}^{d_1}\cdots y_{12}{(m)}^{d_m}$ for all $d_1,\dots ,d_m\ge 0$.

For $g=\left(\begin{array}{cc}\hfill \gamma \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right)\in {\mathrm{\text{GL}}}_2$ with $\gamma \in {𝔽}^{\times }$, we have

for all $\gamma \in {𝔽}^{\times }$. Hence, $\beta =0$, since $𝔽$ is infinite. Therefore, $f({\alpha }_1{E}_{12},\dots ,{\alpha }_m{E}_{12})=\beta =0$. □

Theorem 5.4. Assume that $𝔽$ is infinite. Then

•	$S_{2,m}^{(2)}$, in case $\mathrm{\text{char}}𝔽=2$;
•	$S_{2,m}$, in case $\mathrm{\text{char}}𝔽>2$;

is a minimal separating set for the algebra of invariant polynomial functions on nilpotent matrices $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ for all $m>0$.

Proof. Denote by S the set $S_{2,m}^{(2)}$ or $S_{2,m}$, respectively, in case $\mathrm{\text{char}}𝔽=2$ or $\mathrm{\text{char}}𝔽>2$, respectively. Assume that $\underline{A},\underline{B}\in {\mathcal{𝒩}}_2^m$ are not separated by S and $f\in \mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$. To prove that S is separating we have to show that $f(\underline{A})=f(\underline{B})$. By Remark 5.2, without loss of generality, we can assume that f is ${\mathbb{N}}^m$-homogeneous of multidegree $\underline{\delta }$. Moreover, without loss of generality, we can assume that $f(\underline{A})\ne 0$. By Lemmas 4.2 and 4.4, one of the next two cases holds:

•

$$\begin{array}{ccc}\hfill {\underline{A}}_{\sigma }& \hfill \sim \hfill & (0^{(k+l)},{\alpha }_1{E}_{12},\dots ,{\alpha }_{r+s}{E}_{12}),\hfill \\ \hfill {\underline{B}}_{\sigma }& \hfill \sim \hfill & (0^{(k)},{\beta }_1{E}_{12},\dots ,{\beta }_l{E}_{12},0^{(r)},{\beta }_{l+1}{E}_{12},\dots ,{\beta }_{l+s}{E}_{12}),\hfill \end{array}$$ where $\sigma \in {\mathcal{𝒮}}_m$, $k,l,r,s\ge 0$, ${\alpha }_1,\dots ,{\alpha }_{r+s},{\beta }_1,\dots ,{\beta }_{l+s}\in {𝔽}^{\times }$; moreover, ${\alpha }_1=1$ in case $\underline{A}\ne 0$ and ${\beta }_1=1$ in case $\underline{B}\ne 0$;

•

$\underline{A}\sim \underline{B}$.

In the second case, we have $f(\underline{A})=f(\underline{B})$. Assume that the first case holds.

Define $f_{{\sigma }^{-1}}$ as the result of substitutions $y_{ij}(t)\to y_{ij}({\sigma }^{-1}(t))$ in f for all $1\le t\le m$, $1\le i,j\le 2$. Then $f_{{\sigma }^{-1}}\in \mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ and $f_{{\sigma }^{-1}}({\underline{C}}_{\sigma })=f(\underline{C})$ for all $\underline{C}\in {\mathcal{𝒩}}_2^m$. Therefore, to prove that $f(\underline{A})=f(\underline{B})$ it is enough to show that $f_{{\sigma }^{-1}}({\underline{A}}_{\sigma })=f_{{\sigma }^{-1}}({\underline{B}}_{\sigma })$. Note that ${\underline{A}}_{\sigma }$, ${\underline{B}}_{\sigma }$ are not separated by S by Remark 4.1. Therefore, without loss of generality, we can assume that $\sigma \in {\mathcal{𝒮}}_m$ is the trivial permutation.

If ${\delta }_i\ne 0$ for some $1\le i\le k+l$, then $f(\underline{A})=0$; a contradiction. Otherwise, ${\delta }_1=\cdots ={\delta }_{k+l}=0$. Hence, f is a polynomial in $\{y_{ij}(k+l+1),\dots ,y_{ij}(m)|i,j\in \{1,2\}\}$. Therefore, $f(E_{12}^{(k+l)},{\alpha }_1{E}_{12},\dots ,{\alpha }_{r+s}{E}_{12})=f(\underline{A})$ is non-zero; a contradiction to Lemma 5.3. Thus, S is separating.

Taking $\underline{A},\underline{B}$ from Claims 3, 4 from the proof of Theorem 4.5 we obtain that S is a minimal separating set for all $m>0$. □

6. Corollaries

As in Sec. 1.1, assume that V is an n-dimensional vector space over $𝔽$, G is a subgroup of $\mathrm{\text{GL}}(V)$, and $W\subset V$ is a G-invariant subset of V. We say that an $m_0$-tuple $\underline{j}\in {\mathbb{N}}^{m_0}$ is m-admissible if $1\le j_1<\cdots <j_{m_0}\le m$. For any m-admissible $\underline{j}\in {\mathbb{N}}^{m_0}$ and $f\in \mathcal{𝒪}{(W^{m_0})}^G$ we define the invariant polynomial function $f^{(\underline{j})}\in \mathcal{𝒪}{(W^m)}^G$ as the result of the following substitution of variables in f:

Given a set $S\subset \mathcal{𝒪}{(W^{m_0})}^G$, we define its expansion $S^{[m]}\subset \mathcal{𝒪}{(W^m)}^G$ by

$$S^{[m]}=\{f^{(\underline{j})}|f\in S\mathrm{\text{and}}\underline{j}\in {\mathbb{N}}^{m_0}\mathrm{\text{is}}m\mathrm{\text{-}}admissible\}.$$(13)

Remark 6.1 (Cf. [10, Remark 1.3]). Assume that $S_1$ and $S_2$ are separating sets for $\mathcal{𝒪}{(V^{m_0})}^G$ and assume that $m>m_0$. Then $S_1^{[m]}$ is separating for $\mathcal{𝒪}{(V^m)}^G$ if and only if $S_2^{[m]}$ is separating for $\mathcal{𝒪}{(V^m)}^G$.

Denote by $\sigma \mathrm{\text{sep}}(\mathcal{𝒪}(W),G)$ the minimal number $m_0$ such that the expansion of some separating set S for $\mathcal{𝒪}{(W^{m_0})}^G$ produces a separating set for $\mathcal{𝒪}{(W^m)}^G$ for all $m\ge m_0$. It immediately follows from the main result of [33] that $\sigma \mathrm{\text{sep}}(\mathcal{𝒪}(V),{\mathcal{𝒮}}_n)\le \lfloor \frac{n}{2}\rfloor +1$ over an arbitrary field $𝔽$, where ${\mathcal{𝒮}}_n$ acts on V by the permutation of the coordinates. Moreover, $\sigma \mathrm{\text{sep}}(\mathcal{𝒪}(V),{\mathcal{𝒮}}_n)\le \lfloor {log}_2(n)\rfloor +1$ in case $𝔽={𝔽}_2$ (see [28, Corollary 4.12]).

Corollary 6.2. Assume that $𝔽={𝔽}_q$ is finite and $m\ge 2$. Then

•	$\beta \mathrm{\text{sep}}(\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2})\le 2$, in case $q=2$;
•	$\beta \mathrm{\text{sep}}(\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2})\le 4(q-1)$, in case $q>2$.

Proof. See Theorem 4.5 and formulas (9) and (10). □

Corollary 6.3. Assume that $𝔽$ is infinite and $m\ge 2$. Then

•	$\beta \mathrm{\text{sep}}(\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2})\le 2$, in case $\mathrm{\text{char}}𝔽=2$ or $m=2$;
•	$\beta \mathrm{\text{sep}}(\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2})\le 3$, in case $\mathrm{\text{char}}𝔽\ne 2$ and $m>2$.

Proof. See Theorem 5.4. □

Corollary 6.4. Assume that $𝔽$ is an arbitrary field. Then

Proof. The upper bound on $\sigma \mathrm{\text{sep}}(\mathcal{𝒪}({\mathcal{𝒩}}_2),{\mathrm{\text{GL}}}_2)$ follows from Theorems 4.5 and  5.4. To obtain the lower bound on $\sigma \mathrm{\text{sep}}(\mathcal{𝒪}({\mathcal{𝒩}}_2),{\mathrm{\text{GL}}}_2)$ we consider $\underline{A},\underline{B}$ from Claims 3 and 4 of the proof of Theorem 4.5. □

Corollary 6.5. Assume that $𝔽={𝔽}_q$. Then a minimal separating set

•	$H_{2,m}^{(2)}$, in case $\mathrm{\text{char}}𝔽=2$;
•	$H_{2,m}$, in case $\mathrm{\text{char}}𝔽>2$;

for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ contains the least possible number of elements for a separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ if and only if $m=1$ or $m=q=2$.

Proof. The set from the formulation of corollary is a minimal separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^m)}^{{\mathrm{\text{GL}}}_2}$ by Theorem 4.5. We have

where the binomial coefficient $\left(\genfrac{}{}{0.0pt}{}{m}{k}\right)$ is zero in case $m<k$.

Assume $m=1$. Then Corollary 3.5 implies that $\gamma =|H_{2,m}^{(2)}|=|H_{2,m}|=1$; i.e., the required is proven.

Assume $m=q=2$. Then $\mathrm{\text{char}}𝔽=2$ and Corollary 3.5 implies that $\gamma =|H_{2,m}^{(2)}|=3$; i.e., the required is proven.

Assume $m=2$ and $q\ge 3$. Then Corollary 3.5 implies that $\gamma =2$, but $|H_{2,m}|=|H_{2,m}^{(2)}|=q+1>\gamma$.

Assume $m\ge 3$. Then Corollary 3.5 implies that $\gamma =2m-2$, but $|H_{2,m}|>|H_{2,m}^{(2)}|=m+\left(\genfrac{}{}{0.0pt}{}{m}{2}\right)(q-1)>\gamma$, since $\left(\genfrac{}{}{0.0pt}{}{m}{2}\right)\ge m$. □

Example 6.6. Assume $𝔽={𝔽}_2$ and $m=2$. By Theorem 3.4, each ${\mathrm{\text{GL}}}_2$-orbit on ${\mathcal{𝒩}}_2^2$ contains one and only one element from the following set:

Corollary 6.5 implies that the set

$$H_{2,2}^{(2)}=\{\mathrm{\text{tr}}(Y_1{Y}_2),\zeta (Y_1),\zeta (Y_2)\}$$

is a separating set for $\mathcal{𝒪}{({\mathcal{𝒩}}_2^2)}^{{\mathrm{\text{GL}}}_2}$ containing the least possible number of elements.

Acknowledgment

The work was supported by FAPESP 2018/23690-6.

ORCID

Artem Lopatin  https://orcid.org/0000-0003-2495-6050