Romans 5:5: "and hope does not put us to shame, because God's love has been poured into our hearts through the Holy Spirit who has been given to us."
As with "Savior, When in Dust to Thee," LSB cites a non-existent chapter of Ezra (there are only ten chapters in Ezra). Ezekiel seems to be the intended citation.
Ezekiel 36:26-28: "26 And I will give you a new heart, and a new spirit I will put within you. And I will remove the heart of stone from your flesh and give you a heart of flesh. 27 And I will put my Spirit within you, and cause you to walk in my statutes and be careful to obey my rules. 28 You shall dwell in the land that I gave to your fathers, and you shall be my people, and I will be your God."
John 14:16: "'And I will ask the Father, and he will give you another Helper, to be with you forever,'"
John 14:26: "'But the Helper, the Holy Spirit, whom the Father will send in my name, he will teach you all things and bring to your remembrance all that I have said to you.'"
Romans 15:13: "May the God of hope fill you with all joy and peace in believing, so that by the power of the Holy Spirit you may abound in hope."
+++
The text is public domain:
Come down, O Love divine;
Seek Thou this soul of mine,
And visit it with Thine own ardor glowing;
O Comforter, draw near;
Within my heart appear,
And kindle it, Thy holy flame bestowing.
O let it freely burn,
Till worldly passions turn
To dust and ashes in its heart consuming;
And let Thy glorious light
Shine ever on my sight,
And clothe me round, the while my path illuming.
Let holy charity
Mine outward vesture be
And lowliness become mine inner clothing -
True lowliness of heart,
Which takes the humbler part,
And o'er its own shortcomings weeps with loathing.
And so the yearning strong,
With which the soul will long,
Shall far outpass the pow'r of human telling;
No soul can guess His grace
Till it become the place
Wherein the Holy Spirit makes His dwelling.
+++
The first four citations (Romans 5:5; Ezra 36:26-28; John 14:16, 26) are all fairly similar, and I think they all appear in the first verse, most prominently in the line "Within my heart appear."
The lines "No soul can guess His grace / Till it become the place / Wherein the Holy Spirit makes His dwelling" in the last verse seem to refer to "all joy and peace in believing" from Romans 15:13.
+++
"Thy holy flame bestowing" in the first verse seems to be a reference to Pentecost. Acts 2:1-4: "1 When the day of Pentecost arrived, they [the apostles] were all together in one place. 2 And suddenly there came from heaven a sound like a mighty rushing wind, and it filled the entire house where they were sitting. 3 And divided tongues as of fire appeared to them and rested on each one of them. 4 And they were all filled with the Holy Spirit and began to speak in other tongues as the Spirit gave them utterance."
"My path illuming" in the second verse has some resemblance to Psalm 119:105: "Your word is a lamp to my feet and a light to my path."