Usually, Hagar the Horrible's main residence is a straw-based hatch, but there were various rare occasions where the Viking husband's home was something more grandoise in design.
It's understandable why this large abode would've been changed for something smaller, since all that extra space would be inhibitive for the cartoonist to constantly draw all the details of the exterior. (Which could also explain why these castles were mainly shown at night, since it's less trouble to draw silhouettes than elaborate brick walls) Not to mention that these kinds of castles were normally the focus of viking raids. Living in one of these huge structures would be akin to a family man living in a mansion while going out to conquer high-rise appartments buildings.
While all that extra space a castle provides could prove fertile ground for potential jokes, it could also prove to be an impediment to general storytelling, especially if all that room isn't going to be used effectively with the boundaries of a typical comic page. Sometimes the best devices are the limits we impose upon ourselves. It's unknown exactly when the change from a large castle to modest home was implemented, but this comic below does a pretty good job filling in some gaps that normally wouldn't need explaining.
Chances are, those Lingerons are still living there, long after their purpose has been used.