Both are common. I believe that if you don't repeat the "had" it's understood anyway. I repeat it if there's even the slightest chance of ambiguity. I think in this case I'd be inclined to repeat it, because even though both actions are in the past perfect (precede the simple past "didn't know") the moving may be considered to have occured some time after the marriage.