Problem #1
3y means nothing to the compiler. If you want 3 times y, you have to say
3 * y
Problem #2
A dash is not a minus sign.
Please note, that in the formula "total = 3y + x * (y – a) * (x + b);" you have a dash between y and a, not a minus sign
// – is a dash
// - is a minus sign
// correct formula:
total = 3 * y + x * (y - a) * (x + b);
Problem #3
The variables a and b are not read. You read x and y instead, but you instantly overwrite their value with calculation used by the uninitialized a and b.
Did you mean:
scanf("%f", &a);
scanf("%f", &b);
instead?