The chancel is all that remains of St Andrew's Church in Lilstock. The first church on the site was built before the Norman conquest. In the 12th century the right to appoint a priest was held by the monks of Lonlay in Normandy. In the 15th century this right passed to Eton College. The present church was built in 1532. It consisted of a chancel, nave and tower. By 1554 the chancel was in need of repair and the nave windows were "greatly ruined". In 1557 there was no priest.
The population of Lilstock consisted of 11 households in 1563, 65 taxpayers in 1667, 12 inhabited houses at the end of the 18th century, 56 people in 1801, 91 in 1811, 48 in 1841, 94 in 1881, 58 in 1901 and 5 inhabited houses in 1977.
During the 19th century services were held at St Andrew's Church every Sunday and communion services were held there 4 times a year. The church tower housed 4 bells in 1791 but only 2 by 1881, by which time the church was derelict. In 1881, when the parish of Lilstock was merged with Kilton, the Acland-Hood family had the nave and tower demolished. They had the chancel repaired to serve as a mortuary chapel and a porch and bell-cote added.
The church was declared redundant in 1980 and once again it fell into ruin and was threatened with demolition. The Reverend Rex Hancock, who was the Rector of the United Benefice of Quantoxhead from 1983-93 (which included Lilstock), personally paid for the repair and restoration of St Andrew's Church. The building work was carried out by Arthur Booker, his wife Tilly and his son David and was completed by 1993. Arthur Booker died in 2004 and his ashes are buried in Lilstock churchyard. Rex Hancock died in 2012 aged 84 and is buried at Porlock.
The only 14th century feature still to be seen in the church is the chancel arch, which is just inside the present church doorway. The 12th century font was moved to St Andrew's Church at Stogursey at some point. Sarah Perrett, who was a lady-in-waiting to Queen Victoria and who came from Lilstock, is buried in the churchyard.