Ideas required to solve the problem: 

The general solution of any trigonometric equation is given as –

• sin x = sin y, implies x = nπ + (– 1)ⁿy, where n ∈ Z. 

• cos x = cos y, implies x = 2nπ ± y, where n ∈ Z. 

• tan x = tan y, implies x = nπ + y, where n ∈ Z. 

Given, 

3sin² x – 5 sin x cos x + 8 cos² x = 2 

⇒ 3sin² x + 3 cos² x – 5sin x cos x + 5 cos² x = 2 

⇒ 3 - 5sin x cos x + 5 cos² x = 2 {∵ sin² x + cos²x = 1} 

⇒ 5cos² x + 1 = 5sin x cos x 

Squaring both sides: 

⇒ (5cos² x + 1)² = (5sin x cos x)² 

⇒ 25cos⁴ x + 10cos² x + 1 = 25 sin² x cos² x 

⇒ 25cos⁴ x + 10cos² x + 1 = 25 (1 - cos² x) cos² x 

⇒ 50cos⁴ x – 15 cos² x + 1 = 0 

⇒ 50cos⁴ x – 10 cos² x – 5cos² x + 1 = 0 

⇒ 10cos² x ( 5cos² x – 1) – (5cos² x – 1) = 0 

⇒ (10cos² x - 1)(5cos² x – 1) = 0 

∴ cos² x = 1/10 or cos² x = 1/5 

Hence, when cos² x = 1/10

We have, cos x = ± \(\frac{1}{\sqrt{10}}\)

If cos x = cos y, implies x = 2nπ ± y, where n ∈ Z. 

let cos α = 1/√10 

∴ cos (π – α) = -1/√10 

∴ x = 2nπ ± α or x = 2nπ ± (π – α)

∴ when,  cos x = ± \(\frac{1}{\sqrt{10}}\)

x = 2nπ ± α or x = 2nπ ± (π – α) where n ∈ Z.  and cos α = \(\frac{1}{\sqrt{10}}\)

When cos² x = 1/5

We have, cos x = ± \(\frac{1}{\sqrt{5}}\).

If cos x = cos y, implies x = 2mπ ± y, where n ∈ Z. 

let cos β = 1/√5 

∴ cos (π – β) = -1/√5 

∴ x = 2mπ ± β or x = 2mπ ± (π – β)

∴ when,  cos x = ± \(\frac{1}{\sqrt{5}}\).

x = 2mπ ± β or x = 2mπ ± (π – β)  where n ∈ Z.  and cos β = \(\frac{1}{\sqrt{5}}\)...ans