Matrix Slerp but not around origin?

I'm trying to write a C++ matrix slerp function that will do a decent slerp between two arbitrary matrices to smoothly morph an object between two orientations.

I did this by converting my matrices to quaternions, slerping the quaternions, and converting them back to a final matrix. Works fine as long as the point of rotation is origin. Then I interpolate the translation factors of the two matrices, re-assemble it, and it's all good.

However, in some cases I will be morphing from an identity matrix to a new matrix which might have been translated before rotating. When I do this, although the rotations are correct, I am getting rogue translate because Identity has no translation, but then my rotation has some numbers in the translation area.

How can I factor a non-origin rotation into my slerp?

Edit, adding more information, here is the SPECIFIC problem I'm having at the moment:

1. Green Shape: The object at Identity
2. Yellow Shape: The new position I want to slerp to
3. Cyan Shape: Slerp @ .5f
4. Blue Dot: Position 0,0,0 on the object
5. Red Dot: The offset I am using as a hub

"What I get" is the result when I extract the rotation, slerp the rotation, then interpolate the translation back in. I understand why that's happening-- it's basing the whole slerp off the 0,0,0 position because there's no translation in the final two matrices.

The yellow result is produced by, roughly, this:

OtherMatrix.Translate(0,ysize,0);OtherMatrix.RotateX(80);OtherMatrix.Translate(0,-ysize,0);

Then I do IdentityMatrix.Slerp(OtherMatrix,.5f) which gives me the blue result.

So the question ultimately is, how can I get the left result when I slerp?