Initially, he adopted the Saitō surname but, after Saitō Tatsuoki was decimated by Oda Nobunaga, he changed to the Inoue surname and served Nobunaga.  After the death of Nobunaga in 1582, Yoritsugu entered into service for Toyotomi Hideyoshi, followed by Toyotomi Hideyori.  Later, Yoritsugu, together with his older brother, Inoue Michikatsu, served as members of the kiboro group, an elite division of mounted cavalry soldiers in the Toyotomi army who deployed yellow capes while riding their horses to deflect arrows or other objects hurled at them.  In 1614, Yoritsugu was assigned to a fortress in Shigino to the northeast of Ōsaka Castle with a contingent of 2,000 soldiers in an arquebus battalion.  On 11/26 of Keichō 19 (1614), at the Battle of Shigino, Yoritsugu was killed in action after a fierce attack by the Edo bakufu army led by Uesugi Kagekatsu.  This was one battle in the Winter Campaign of the Siege of Ōsaka between the Edo bakufu army and the Toyotomi army.