In 1820, Wilhelm Duke of Nassau gave part of the former castle grounds to the Catholic community for the construction of a church. However, as the land was swampy, the parish sold it to the master roofer Konrad Kalb from Lorch am Rhein in 1826.

Kalb built a simple three-storey building in the neoclassical style, which was very large for a private house and not only served as a residence, but also housed the "Zum deutschen Haus" inn. Kalb also ran a grocer's store and a building materials store. Rooms were also rented to Stephan Herber, who set up the Hirsch pharmacy there.

During the construction of the city palace, it became apparent that it was too small for a larger court. Duke Wilhelm therefore acquired the Kalb estate shortly before his death in 1839. His successor, Duke Adolph zu Nassau, had the remains of the old castle still standing in the courtyard demolished and built the riding arena and the stables building there. The Kalb'sche Doppelhaus housed the ducal court marshal's office and has been known as the Kavalierhaus ever since.

During the Second World War, the Kavalierhaus was so badly destroyed that it could not be rebuilt. The city therefore launched an architectural competition with the aim of constructing an office building for the staff of the Federal Statistical Office, which was to have one more storey than the Kavalierhaus. The style of the new building was based on the late classicist design of the previous building. The offices of the Federal Statistical Office were able to move in in 1953.

The building has been used by the Hessian State Parliament since 1975.