The library was founded in 1928, when it was located on the second floor of the Pawling Firehouse. Today, Pawling Free Library occupies two buildings: its main building was renovated in 2002; most recently, the Library acquired the "Yellow House" next door, and currently uses this building for children's programs and meetings.