The third verse ends with the lines "Ascribing pow'r and glory / And praise to God alone." This echoes the first two verses of Psalm 29: "1 Ascribe to the LORD, O heavenly beings, ascribe to the LORD glory and strength. 2 Ascribe to the LORD the glory due his name; worship the LORD in the splendor of holiness."
In the verse for St. Matthew, the lines "From all unrighteous mammon, / O raise our eyes anew / That we in our vocation / May rise and follow You" (particularly the last line) come from Matthew 9:9: "As Jesus passed on from there, he saw a man called Matthew sitting at the tax booth, and he said to him, 'Follow me.' And he rose and followed him."