Henry Francis Lyte moved to All-Saints Church in Brixham, Devon in 1824, where he became chairman of the schools committee, established the first Sunday school in the Torbay area and created a Sailors' Sunday School. The primary object of both schools was to provide education for children and seamen for whom other schooling was almost impossible. He organised an Annual Treat for the 800-1000 Sunday school children, which included a short religious service followed by tea and sports in the field. Shortly after Lyte's arrival in Brixham, he attracted such large crowds that the church had to be enlarged.

Lyte was an expert flute player, spoke Latin, Greek, and French; enjoyed discussing literature; and was knowledgeable about wild flowers. At his Brixham home, Berry Head House, a former military hospital, Lyte created a magnificent library largely of theology and old English poetry, described in his obituary as one of the most extensive and valuable in the West of England.

Nevertheless, Lyte was also able to identify with his parish of fishermen, visiting their homes and their ships in harbour, supplying every vessel with a Bible, and compiling songs and a manual of devotions for use at sea. A friend of Samuel Wilberforce, he also opposed slavery, organising an 1833 petition to Parliament requesting it be abolished in Great Britain.

In poor health throughout his life, Lyte suffered various respiratory illnesses including asthma and bronchitis, and by the 1840s, he was spending much of his time in the warmer climates of France and Italy. Lyte spent the summer of 1847 at Berry Head, writing his best known hymn, Abide With Me. After one final sermon to his congregation he left again for Italy, and died at Nice on 20 November 1847. Other well-known hymns include Praise, my Soul, the King of Heaven and Pleasant are Thy Courts Above.