I noticed today that Paul says a very similar thing in both Philippians and 2 Corinthians in describing his pastoral labors.
Phil 1:25 - I will remain and continue with all of you for your progress and joy of faith.
2 Cor 1:24 - We work with you for your joy.
His goal was his peoples' happiness. He prayed that the Colossians would be strenghened with all endurance and patience with joy (1:11).
Worth reflecting on. I acknowledge and agree that the glory of God is the highest purpose in everything, the supreme goal of all we do, whether pastors or not. But I wonder if on a horizontal level there could be any higher purpose in pastoral work than the pursuit of the joy of those in our sphere of influence (and that, yes, this is a fundamental way in which God is in fact glorified).