Aggregation algorithms that provide services to the grid via demand side management are moving from research ideas to the market. With the diversity of the technology delivering such services, it becomes essential to establish transparent performance standards from a service delivery perspective. This paper formulates performance measures and an index to evaluate in hindsight the quality of service delivery by an aggregator, both with respect to ancillary service and asset management service. The index is based on requirements formulated in service contracts and provides an overall assessment of the quality of service provided by an aggregation control algorithm. By a detailed case study we present and an application of the index, comparing the performance of two different control architectures for demand side management delivering a distribution grid service.