(D² -5D + 6)y = x sin3x.

It's auxiliarly equation is,

m² - 5m + 6 = 0

⇒ (m-2)(m-3) = 0

⇒ m = 2 or 3 (real and distinct)

Therefore,

C. F = C₁e^2x + C₂e^3x

P.I = \(\frac{1}{D^2-5D+6}x\,sin\,3x\)

=  \(x\frac{1}{D^2-5D+6}\,sin\,3x\) + {\(\frac{d}{dD}\,\)\(\frac{1}{D^2-5D+6}\)} sin3x

(∵ \(\frac{1}{f(D)}\)xv = x\(\frac{1}{f(D)}\)v + {(\(\frac{d}{dD}\,\) \(\frac{1}{f(D)}\))v} and here v = sin3x)

= x\(\frac{1}{-9-5D+6}\) sin3x - \(\frac{2D-5}{(D^2-5D+6)^2}\) sin3x

(∵ \(\frac{d}{dx}\) \(\frac{1}{x}\) = \(\frac{-1}{x^2}\) and by chain rule Also we use\(\frac{1}{f(D^2)}\)sinax = \(\frac{1}{f(-a^2)}\)sinax)

= - x\(\frac{1}{3+5D}\)sin3x - \(\frac{2D-5}{(-9-5D+6)^2}\)sin3x

= -x \(\times\) \(\frac{3-5D}{9-25D^2}\)sin3x - \(\frac{2D-5}{(3+5D)^2}\)sin3x

= \(\frac{-x(3-5D)sin3x}{9-25\times -9}\) - \(\frac{2D-5}{25D^2+30D+9}\)sin3x

= \(\frac{x}{234}\)(3 sin3x - 5D sin3x) - \(\frac{2D-5}{25\times{(-9)}+30D+9}\)sin3x

= \(\frac{x}{234}\)(3 sin3x - 15 cos3x) - \(\frac{2D-5}{30D-216}\)sin3x

(∵ D sin3x = \(\frac{d}{dx}\) sin3x = 3cos 3x)

= \(\frac{3x}{234}\)(sin3x - 5cos3x) - \(\frac{1}{6}\times\)\(\frac{(2D-5)(5D+36)}{(5D-36)(5D+36)}\)sin3x

= 78x (sin3x - 5cos3x) - \(\frac{1}{6}\times\)\(\frac{(10D^2+47D-180)}{25D^2-1296}\)sin3x

=  78x (sin3x - 5cos3x) - \(\frac{1}{6}\times\) \(\frac{10D^2\,sin3x + 47D\,sin3x-180\,sin3x}{25\times -9-1296}\)

=  78x (sin3x - 5cos3x) - \(\frac{1}{6}\times\) \(\frac{30D\,cos3x +141\,cos3x-180\,sin3x}{1521}\)

=  78x (sin3x - 5cos3x) + \(\frac{1}{9126}\) (-90 sin3x + 141cos3x - 180sin3x)

=  78x (sin3x - 5cos3x) + \(\frac{1}{9126}\)(141 cos3x - 270 sin3x)

Hence,

complete solution given diff.equation is 

I = C.F + P.I

= C₁e^2x + C₂e^3x + 78x(sin3x - 5cos 3x) + \(\frac{1}{9126}\)(141cos3x - 270 sin3x).