Đáp án:

$S = \{3; 8\}$

Giải thích:

ĐKXĐ: $x \ge -1$

$2x^2 - 6x + 10 - 5(x - 2)\sqrt{x + 1} = 0$

$\Leftrightarrow 2(x - 2)^2 + 2(x + 1) - 5(x - 2)\sqrt{x + 1} = 0$

$\Leftrightarrow \left[2(x - 2)^2 - (x - 2)\sqrt{x + 1}\right] + \left[2(\sqrt{x + 1})^2 - 4(x - 2)\sqrt{x + 1}\right] = 0$

$\Leftrightarrow (x - 2)\left[2(x - 2) - \sqrt{x + 1}\right] - 2\sqrt{x + 1}\left[2(x - 2) - \sqrt{x + 1}\right] = 0$

$\Leftrightarrow \left[2(x - 2) - \sqrt{x + 1}\right]\left[(x - 2) - 2\sqrt{x + 1}\right] = 0$

$\Leftrightarrow \left[\begin{aligned} 2(x - 2) - \sqrt{x + 1} = 0 \\ (x - 2) - 2\sqrt{x + 1} = 0 \end{aligned}\right.$

$\text{TH1:}$ $2(x - 2) - \sqrt{x + 1} = 0$

$\Rightarrow 2(x - 2) = \sqrt{x + 1}$

ĐK: $x - 2 \ge 0 \Leftrightarrow x \ge 2$

$4(x - 2)^2 = x + 1$

$\Leftrightarrow 4(x^2 - 4x + 4) = x + 1$

$\Leftrightarrow 4x^2 - 17x + 15 = 0$

$\Leftrightarrow (x - 3)(4x - 5) = 0$

$\Leftrightarrow \left[\begin{aligned} x - 3 = 0 \\ 4x - 5 = 0 \end{aligned}\right.$$\Leftrightarrow \left[\begin{aligned} x = 3 \\ 4x = 5 \end{aligned}\right.$$\Leftrightarrow \left[\begin{aligned} x = 3 \text{ (TM)} \\ x = \dfrac{5}{4} \text{ (Loại)} \end{aligned}\right.$

$\Rightarrow x = 3$

$\text{TH2:}$ $(x - 2) - 2\sqrt{x + 1} = 0$

$\Rightarrow x - 2 = 2\sqrt{x + 1}$

ĐK: $x - 2 \ge 0 \Leftrightarrow x \ge 2$

$(x - 2)^2 = 4(x + 1)$

$\Leftrightarrow x^2 - 4x + 4 = 4x + 4$

$\Leftrightarrow x^2 - 8x = 0$

$\Leftrightarrow x(x - 8) = 0$

$\Leftrightarrow \left[\begin{aligned} x = 0 \\ x - 8 = 0 \end{aligned}\right.$$\Leftrightarrow \left[\begin{aligned} x = 0 \text{ (Loại)} \\ x = 8 \text{ (TM)} \end{aligned}\right.$

$\Rightarrow x = 8$

Đối chiếu ĐKXĐ $x \ge -1$, cả hai nghiệm đều TM

Vậy $S = \{3; 8\}$