In 1606, upon the death of his grandfather, Kuwayama Shigeharu, Kiyoharu inherited a fief of 10,000 koku out of a retirement allowance of 16,000 koku.
In 1609, however, he incurred the wrath of the bakufu and wad forced to live in confinement. As a result, the Tanagawa domain was extinguished and his landholdings annexed by the Gose domain led by his father, Motoharu. The Gose domain held territory in the Katsujō and Katsuge districts of Yamato Province and a portion of the Hine District in Izumi Province.