Cloud computing is a new technology that has gained a lot of attention. Software as a Service (SaaS) is using the Cloud technology to deliver software to users. One of the main issues with different SaaS configurations is measuring and comparing the performance and scalability. Existing work in this area typically use multiple metrics and different measurement scales, which makes it hard to compare alternative configurations. This paper proposes a new approach to measure and evaluate the performance and scalability of a SaaS by comparing the cost of running the software on the Cloud. A case study is presented to demonstrate the viability of the approach.