Psalm 119:10: "With my whole heart I seek you; let me not wander from your commandments!"
Ephesians 1:7: "In him we have redemption through his blood, the forgiveness of our trespasses, according to the riches of his grace"
2 Timothy 1:8-10: "8 Therefore do not be ashamed of the testimony about our Lord, nor of me his prisoner, but share in suffering for the gospel by the power of God, 9 who saved us and called us to a holy calling, not because of our works but because of his own purpose and grace, which he gave us in Christ Jesus before the ages began, 10 and which now has been manifested through the appearing of our Savior Christ Jesus, who abolished death and brought life and immortality to light through the gospel"
John 8:31: "So Jesus said to the Jews who had believed in him, 'If you abide in my word, you are truly my disciples'"
+++
The text is public domain:
Let me be Thine forever,My faithful God and Lord;Let me forsake Thee neverNor wander from Thy Word.Lord, do not let me waver,But give me steadfastness,And for such grace foreverThy holy name I'll bless.Lord Jesus, my salvation,My light, my life divine,My only consolation,I make me wholly Thine!For Thou hast dearly bought meWith blood and bitter pain.Let me, since Thou hast sought me,Eternal life obtain.And Thou, O Holy Spirit,My comforter and guide,Grant that in Jesus' meritI always may confide,Him to the end confessingWhom I have known by faith.Give me Thy constant blessingAnd grant a Christian death.
+++
Psalm 119:10 and John 8:31 both seem to be present in the lines "Let me forsake Thee never / Nor wander from Thy Word" (and perhaps even extending into "Lord, do not let me waver, / But give me steadfastness") in the first verse. There's a stronger resemblance between "Nor wander from Thy Word" and "let me not wander from your commandments" from Psalm 119:10. "'Abide in my word'" from John 8:31 has basically the same the meaning, just lookt at from a different perspective.
Ephesians 1:7 appears in the second verse: "For Thou hast dearly bought me / With blood and bitter pain."
2 Timothy 1:8-10 seems to bridge the first and second verses, with verse 8 ("Therefore do not be ashamed of the testimony about our Lord...") appearing at the end of the hymn's first verse ("Lord, do not let me waver, / But give me steadfastness") and verses 9-10 ("who saved us and... brought life and immortality to light through the gospel") at the beginning of the hymn's second verse ("Lord Jesus, my salvation, / My light, my life divine, / My only consolation").