The original white wooden building which housed a reading room, game room, and books and newspapers, was owned by Mrs. J. P. Morgan. She lived in the village in summer months, and local residents were quite fond of her. In about 1884, she "gave" the library to the village, and continued to support it financially. It was chartered by the State in 1896. In 1923, she sold the building to the local library association for ten dollars.

The white building burned down in 1952. At that time it also housed an electrical appliance/repair shop, and the narrow section of the building on the north side held the small collection. In 1952, shortly after reorganizing itself, the library board was able to rebuild on the same property, and so the village now had a red brick building, with a complete children's room in the basement. In the 1990's the trustees began a private fund-raising campaign to add onto the red building, and by August 6, 2001, the new white cottage-Victorian building was opened. The architect/designer was Conrad Remick of New Windsor, NY, and the contractor was John Flannery of Rock Tavern, NY. Monies were raised through donations of corporations, foundations, and private citizens of the Town of Highlands. A mortgage was taken at the end of the project to finish paying expenses, and donations are still gratefully accepted. There are several ways to acknowledge donations within the new building.