a+b+c=1, so c=1-a-b.
a²+b²+(1-a-b)²=2
a²+b²+1-2a-2b+a²+2ab+b²=2
2a²+2b²-2a-2b+2ab=1 ①
a³+b³+(1-a-b)³=3
a³+b³+1-3a-3b+3a²+6ab+3b²-a³-b³-3a²b-3ab²=3
-3a-3b+3a²+6ab+3b²-3a²b-3ab²=2 ②
Rearrange ① and ②:
-2a-2b+2a²+2b²+2ab=1 ①
-3a-3b+3a²+3b²-3a²b-3ab²+6ab=2 ②
Multiply equation ① by -3/2:
3a+3b-3a²-3b²-3ab=-3/2
and add to ②:
-3a²b-3ab²+3ab=½.
So, 3ab(-a-b+1)=½.
But c=1-a-b, so 3abc=½ and abc=⅙.