On existence and phase separation of solitary waves for nonlinear Schrödinger systems modelling simultaneous cooperation and competition

We study existence and phase separation, and the relation between these two aspects, of positive bound states for the nonlinear elliptic system $$\begin{aligned} \left\{ \begin{array}{l@{\quad }l} - \Delta u_i + \lambda _i u_i = \sum \nolimits _{j=1}^d \beta _{ij} u_j^2 u_i &{} \hbox { in }\, \Omega \\ u_1 =\cdots = u_d=0 &{} \hbox { on }\, \partial \Omega . \end{array}\right. \end{aligned}$$-Δui+λiui=∑j=1dβijuj2uiinΩu1=⋯=ud=0on∂Ω.This system arises when searching for solitary waves for the Gross–Pitaevskii equations. We focus on the case of simultaneous cooperation and competition, that is, we assume that there exist two pairs $$(i_1,j_1)$$(i1,j1) and $$(i_2,j_2)$$(i2,j2) such that $$i_1 \ne j_1$$i1≠j1, $$i_2 \ne j_2$$i2≠j2, $$\beta _{i_1 j_1}>0$$βi1j1>0 and $$\beta _{i_2,j_2}<0$$βi2,j2<0. Our first main results establishes the existence of solutions with at least $$m$$m positive components for every $$m \le d$$m≤d; any such solution is a minimizer of the energy functional $$J$$J restricted on a Nehari-type manifold$$\mathcal {N}$$N. At a later stage, by means of level estimates on the constrained second differential of $$J$$J on $$\mathcal {N}$$N, we show that, under some additional assumptions, any minimizer of $$J$$J on $$\mathcal {N}$$N has all nontrivial components. In order to prove this second result, we analyse the phase separation phenomena which involve solutions of the system in a not completely competitive framework.