Up the hill close to the airport is the Paleocastro, or Acropolis.
The ruins may not be of great interest to many but you can scramble around for a good hour. Probably the best time to go is just before sunset as the vantage point is superb for sunset photos over the surrounding small islands and sea.