`(3a^2+7b^2)a+(7a^2+3b^2)b=4+16ab`

`3a^3+7ab^2+7a^2b+3b^3=4+16ab`

`(3a^3+3b^3)+7ab^2+7a^2b=4+16ab`

`3(a^3+b^3)+7ab(a+b)=4+16ab`

`3(a+b)(a^2-ab+b^2)+7ab(a+b)=4+16ab`

`(a+b)[3(a^2-ab+b^2)+7ab]=4+16ab`

`(a+b)[3(a^2+b^2)-3ab+7ab]=4+16ab`

`(a+b)(3.2+4ab)=4+16ab`

`(a+b)(6+4ab)=4+16ab`

`6(a+b)+4ab(a+b)=4+16ab`

Đặt `x=a+b;y=ab`, ta có:

`6x+4xy=4+16y`

Mặt khác `a^2+b^2=2`

`(a+b)^2-2ab=2`

`x^2-2y=2`

`2y=x^2-2`

`y=(x^2-2)/2`

Thay `y=(x^2-2)/2`, ta có:

`6x+4x . (x^2-2)/2=4+16 . (x^2-2)/2`

`6x+2x(x^2-2)=4+8(x^2-2)`

`6x+2x^3-4x=4+8x^2-16`

`2x^3+2x-4-8x^2+16=0`

`2x^3-8x^2+2x+12=0`

`x^3-4x^2+x+6=0`

`x^3+x^2-5x^2-5x+6x+6=0`

`x^2(x+1)-5x(x+1)+6(x+1)=0`

`(x^2-5x+6)(x+1)=0`

`(x^2-3x-2x+6)(x+1)=0`

`[x(x-3)-2(x-3)](x+1)=0`

`(x-2)(x-3)(x+1)=0`

`x-2=0` hoặc `x-3=0` hoặc `x+1=0`

`x=2` hoặc `x=3` hoặc `x=-1`

Với `x=2=>y=(2^2-2)/2=1`

Với `x=3=>y=(3^2-2)/2=7/2`

Với `x=-1=>y=((-1)^2-2)/2=-1/2`

Th1: `x=2;y=1`, ta có:

`{(ab=1),(a+b=2):}`

Ta có: `a+b=2=>a=2-b`

Thay `a=1-b`, ta có:

`(2-b)b=1`

`-b^2+2b-1=0`

`b^2-2b+1=0`

`(b-1)^2=0`

`b-1=0`

`b=1`

`=>a=2-1=1`

Th2: `x=3;y=7/2`, ta có:

`{(a+b=3),(ab=7/2):}`

Ta có: `a+b=3=>a=3-b`

Thay `a=3-b`, ta có:

`(3-b)b=7/2`

`2(3b-b^2)=7`

`-2b^2+6b-7=0`

`2b^2-6b+7=0`

`Delta'=(-3)^2-2.7=-5<0=>` pt vô nghiệm

Th3: `x=-1;y=-1/2`, ta có:

`{(a+b=-1),(ab=-1/2):}`

Ta có: `a+b=-1=>a=-b-1`

Thay `a=-b-1`, ta có:

`(-b-1)b=-1/2`

`(b+1)b=1/2`

`2(b^2+b)=1`

`2b^2+2b-1=0`

`Delta'=1^2-2.(-1)=3>0=>` pt có `2` nghiệm pb

`b_1=(-b'+\sqrt{Delta'})/a=(-1+\sqrt{3})/2=>a_1=-(-1+\sqrt{3})/2-1=(1-\sqrt{3}-2)/2=(-1-\sqrt{3})/2`

`b_2=(-b'-\sqrt{Delta'})/a=(-1-\sqrt{3})/2=>a_2=-(-1-\sqrt{3})/2-1=(1+\sqrt{3}-2)/2=(-1+\sqrt{3})/2`

Vậy `(a;b)={(1;1);((-1-\sqrt{3})/2;(-1+\sqrt{3})/2);((-1+\sqrt{3})/2;(-1-\sqrt{3})/2)}`