April 26, 1999

TO:City of Toronto Audit Committee

FROM:Norman Gardner, Chairman

Toronto Police Services Board

SUBJECT:RESULTS OF THE CITY OF TORONTO AUDIT ON THE TORONTO POLICE SERVICE METROPOLIS PROJECT & RESPONSE

Recommendations:

It is recommended that:

1.That the City of Toronto Audit Committee receive the following report for information; and

2.That the City of Toronto Audit Committee forward this report to the next meeting of Council for information.

Council Reference/Background History:

At its meeting on March 26, 1999, the Toronto Police Services Board was in receipt of the following report MARCH 1, 1999 from Jeffrey Griffiths, City Auditor, City of Toronto:

"Subject:Review of Metropolis Project

Recommendation:

It is recommended that:

1.Project benefits calculations/projections be reviewed by Police Services management for validity and reasonableness prior to incorporation into any report submitted to the Board and/or Council;

2.Where projects and related benefits span several years, the cost savings recognized in prior years be reviewed annually to ensure that such benefits may still appropriately be included in any cumulative annualized benefits calculations;

3.Clear parameters be established at the beginning of each project, identifying its scope and the benchmarks to be used in evaluating the success of the project. Such benchmarks must be relevant, objective and measurable;

4.Where projects are approved on the basis of efficiency savings, a plan should be established outlining how staff redeployments will be achieved, and a reporting process be developed to identify such redeployments so that they may be properly reflected in future operating budgets. In the event that the efficiency savings will be absorbed by increases in service levels or demands, these increases should be clearly identified;

5.The City of Toronto, Budget Division, clarify to all Departments, Agencies, Boards and Commissions that the Impact on the Operating Budget section of capital budget submissions be restricted to actual budget reductions;

6.Future submissions to the Toronto Police Services Board and Council clearly identify the types of benefits to be realized (i.e. actual dollars vs efficiency savings);

7.Police Service staff investigate the possibility of incorporating project accounting capabilities into any future financial systems; and

8.Any cost/benefit analysis performed in support of projects should include both external as well as internal costs.

Background:

In 1991 the Metropolis (Metropolitan Toronto Police Information System) strategy was conceived to develop an integrated computing environment to improve the efficiency, flexibility and effectiveness of the Police Service's business systems in support of its Beyond 2000 policing strategy. A critical component of the Beyond 2000 strategy was Community Policing, which in general terms involved the movement of police officers out of cars and offices onto the streets. Prior to Metropolis the Police still had many manually intensive processes including significant use of typewriters and manual paper files. The Service identified that a major technology initiative was required to improve the level of Police services in the City of Toronto. Metropolis was the name given to this initiative.

Metropolis involved the development of hardware and software architectures which allow for an extremely high degree of compatibility and interconnectivity across the Service. It also encompassed the development of a number of operational systems in a broad spectrum of police operations. Data previously collected and stored in various forms, plus other data considered pertinent to operations, is now captured and stored electronically.

Between 1992 and 1997, the former Metro Council approved capital funding for Metropolis of approximately $60 million with the expectation that the project would deliver a cumulative net benefit in excess of $90 million by the end of the year 2002. For the Police Service the primary benefit was to allow for an improvement in services to the citizens of Toronto funded through the efficiency gains to be delivered by the Metropolis initiatives.

Comments:

At its meeting of May 21, 1998, the Toronto Police Services Board requested the City Auditor to perform a value for money audit of the expenditures incurred with respect to the Metropolis project.

The objective of this review was to determine, to the extent possible, whether the benefits realized from the Metropolis project justified the expenditures incurred.

The review included interviews with various staff from the Police Services and an examination of relevant records and documents covering the period 1992 to 1998, including:

·meeting minutes of the former Metro Council, Police Services Board, Metropolis Advisory Council and Project Review Committee;

·operating and capital budget submissions;

·internal cost records relating to Metropolis maintained by the Service;

·purchase orders and accounts payable vouchers;

·benefits calculations and user sign off's;

·budgets and workplans of specific projects;

·previous audit reports.

Metropolis Benefits

The former Metro Council approved the Metropolis project in 1992 on the basis of operational savings that were projected to be realized from the implementation of Metropolis.

The Computing and Telecommunications Unit of the Police (C&T) in conjunction with user departments performed annual studies, (referred to as Metropolis Benefits Studies), to determine the benefits realized from the Metropolis project. The benefits were classified into three categories: actual dollar savings; cost avoidances and productivity gains, (grouped together as efficiency savings below). A dollar value was calculated for every benefit. The following table summarizes the benefits identified by Police Services for the years 1993 to 1997:

Cumulative

Actual EfficiencyTotalGross Annual

YearDollar Savings Savings Savings Savings

1993 $42,500$1,699,500$1,742,000 $1,742,000

1994 3,361,50016,208,50019,570,00021,312,000

1995 107,000 7,748,000 7,855,000$29,167,000

1996 328,00010,343,00010,671,000$39,838,000

1997 100,000 7,695,000 7,795,000$47,633,000

Total $3,939,000$43,694,000$47,633,000 $139,692,000

According to these internal studies, the Service has realized cumulative net benefits totalling approximately $80 million ($139.7 million benefits less $60 million in costs) as of the end of 1997, and will realize from 1998 onwards annual benefits of $47.6 million. This is well ahead of their initial projections of $90 million in cumulative net benefits by the year 2002.

As previously noted, the savings listed in the table above represent amounts identified by staff of the Police Services. The vast majority of the savings arise from increased efficiencies which have been attributed dollar values based on various estimates and assumptions. To accurately measure efficiency savings it is necessary to assess how work is performed both before and after any change. However, this type of bench marking exercise was not undertaken prior to project implementation and it is difficult, if not impossible, to verify the claimed savings. Our review of these savings indicated that there was no conclusive evidence that the actual dollar savings were actually achieved. Having said that, however, there is evidence that there were efficiency savings which either enabled or assisted the Police Service to endure budget reductions during the five-year period noted above.

During that period the operating budget of the Police Service declined from $560 million in 1993 to $495 million in 1997, a decrease of $65 million. In addition, during the same time frame there was a reduction of 513 in the number of staff at the Service.

Following are our comments on various components of the savings listed above and suggestions for improvements to future major capital projects which should improve the ability to measure savings realized.

Actual Dollar Savings

Of the $47.6 million annual benefits, $3.9 million (8%) represents actual dollar savings and $43.7 million (92%) represents efficiency savings and productivity gains. The majority of the actual dollar savings of $3.9 million is attributed to the implementation of Court Scheduling System (CASC) .

The objective of the CASC system is to improve the scheduling of police officers appearing in court for non-criminal cases by supplying officer available dates to the Ministry of the Attorney General's Office (MAG). These dates are supplied so that cases will not be scheduled when an officer is off duty, thus reducing officer overtime for court attendance. However, the supplied dates are used solely at the discretion of MAG and the Police Court Services Unit has no control in ensuring that MAG uses the CASC supplied available dates. A review of court appearance statistics supplied by Court Services indicate that historically, the usage rate of CASC supplied dates is only approximately 35 percent.

The 1994 Metropolis Benefits Study reported that the CASC system was directly responsible for a $3.25 million permanent reduction in non criminal court overtime cost with an additional $135,000 being saved in other miscellaneous areas. This was based on the fact that the budget for premium pay was reduced by $12 million from 1992 to 1994. In reviewing the 1994 budget requirements, the Police Service determined that it did not require the funding commitment of previous years due to various programs it had implemented, Metropolis being one of them. Therefore, Police staff considered it reasonable that almost 25 percent ($3.25 million) of the overall budget reduction in premium pay could be attributed to the Metropolis implementation.

While the $3.25 million estimate noted above may be reasonable it is highly unlikely that these savings were all actual dollar savings. A review of the general ledger accounting records noted that although the budget for total premium pay was reduced by $12 million from 1992 to 1994, actual premium pay dropped by only $3.6 million over the same period. By claiming that $3.25 million, (90%), of this reduction is related to non criminal court overtime versus other forms of overtime costs, may be overstating the savings attributed to the CASC system given other initiatives that were underway to reduce overall premium pay within the Service. Further, in an internal memo dated April 22, 1996, the Court Services Office estimated savings with respect to non criminal court costs to be $1.2 million per year as opposed to the $3.25 million.

In addition, other factors may have also contributed to the decrease in court overtime costs. For example, during this period when premium pay expenses were decreasing, the number of uniform officers and enforcement figures were also showing a steady decline. On a per capita basis, the amount of premium pay per officer has remained relatively constant since 1992. This could mean two things in terms of the Metropolis project. First, Metropolis initiatives allowed the Service to reduce staff and still provide an adequate level of policing to the City. Or alternatively, the reduction in Policing costs was purely due to a reduction in the number of officers on the street and the concomitant reduced level of police service provided in the City. Again, there is little hard evidence to support either of these theories although the magnitude of the budget reductions during this period does suggest that efficiency savings allowed the Service to absorb the budget cuts without significant reductions in service.

Although not significant in the overall analysis of the Metropolis project, our review of the actual dollar savings as identified by the Service noted a flaw in the calculations presented in the 1996 Metropolis Benefits Study. The 1996 Study attributed to CASC a further reduction reflected in the Police budget of $298,000 in non criminal court overtime costs. This was based on a comparison of actual court overtime costs in the first quarter of 1995 compared to the first quarter of 1996 and projecting the savings for an entire year. What does not appear to have been considered is that in early 1996, an OPSEU strike put a number of court cases on hold, which may have accounted for the reduction in court overtime costs in the first quarter of 1996. Our review of court overtime costs for the entire 1996 year indicated that expenses actually increased by $120,000 over 1995 and it is therefore very difficult to justify the claimed actual annual savings of $298,000.

In our examination of court overtime costs as reflected in the general ledger, it was noted that since 1995, court overtime costs have shown a steady increase and that these costs have consistently exceeded budget. However, the Police accounting system does not accurately identify court or overtime costs. As such it is almost impossible to verify that savings were actually achieved from these initiatives.

Information for inclusion in the benefits studies came from various Units within the Service. A more rigorous review by management of the information included in the benefits studies may have recognized the deficiencies we have identified in these studies. If these deficiencies had been identified by management further analysis could have been conducted.

Finally, there is a potential for the marketing to other Police Services of the Metropolis technology. This has been an issue raised by the City Auditor previously. We understand that the Police have recently initiated a marketing strategy pertaining to Metropolis. To date the Service has made recoveries of approximately $150,000 on sale of technology related to Metropolis. The sale of Metropolis technology has the potential to recover some of the upfront investment associated with the project.

Recommendations:

It is recommended that:

1.Project benefits calculations/projections be reviewed by Police Services management for validity and reasonableness prior to incorporation into any report submitted to the Board and/or Council; and

2.Where projects and related benefits span several years, the cost savings recognized in prior years be reviewed annually to ensure that such benefits may still appropriately be included in any cumulative annualized benefits calculations.

Efficiency Savings and Productivity Gains

The Metropolis Advisory Council (MAC) consisting of industry experts, Toronto Police staff and Metropolitan Officials was formed to provide an independent assessment of the technology strategy, approach and architecture of Metropolis. In its inaugural meeting on October 9, 1992, it was pointed out that the benefits to be derived from the Metropolis project were likely to be time-savings rather than actual dollar savings.

The quantification of time savings was for the purposes of performing a cost/benefit analysis for project justification and was not meant to be interpreted as actual budget reductions. The time savings calculations were based on intuitive estimates versus a formal study. A dollar amount was assigned by multiplying the time savings by the number of working days and the annual average salary/benefits cost for Uniform and Civilian staff.

Our analysis of the efficiency benefits calculations indicated that the three projects realizing the largest efficiency benefits were the Office Tools, Alternate Response Unit and Voice Mail projects, accounting for $23.8 million (54%) of the $43.7 million of total annual efficiency savings.

It was noted that the savings attributed to the Office Tools and Voice Mail projects were in small increments of 5 to 15 minutes per day extrapolated across the Service. The time savings appear reasonable, however, multiplying the time savings by the labour costs may overstate their value as time savings in such small quantities would generally not be re-deployable. A more relevant benchmark would have been to identify specific redeployments of staff associated with each Metropolis project, indicating where resources were moved to. However, no process was established to track staff reductions/redeployments and as such it is impossible to ascertain how many, if any of the positions eliminated were attributed to Metropolis versus other factors/initiatives (restructuring, delayering, civilianisation of positions, etc.).

Because the quantification/valuation of efficiency benefits is so subjective and the fact that appropriate, relevant and measurable benchmarks were not established at the beginning of the project to allow for a proper evaluation, it was impossible to substantiate the efficiency benefits reported. This is not to say that all benefits should be quantified in financial terms. There may be instances where quantification of benefits in finite dollar terms becomes so subjective as to negate the value of the quantification exercise. In these circumstances it may be necessary to enumerate the benefits without assigning a specific dollar value to them using instead a different, non financial, measure to determine how project success will be evaluated. It then becomes a matter of judgement as to whether or not the initiative giving rise to the benefit should be undertaken. The key is there must be a defined benefit that will be achieved by the project and there must also be a method of measuring success.

Recommendations:

It is recommended that:

3.Clear parameters be established at the beginning of each project, identifying its scope and the benchmarks to be used in evaluating the success of the project. Such benchmarks must be relevant, objective and measurable; and

4.Where projects are approved on the basis of efficiency savings, a plan should be established outlining how staff redeployments will be achieved, and a reporting process be developed to identify such redeployments so that they may be properly reflected in future operating budgets. In the event that the efficiency savings will be absorbed by increases in service levels or demands, these increases should be clearly identified.

Capital Budget Submissions

The Metropolis project required annual approval from the Toronto Police Services Board and Metro Council. Capital budget submissions were made each year, detailing the progress and benefits accumulated to date and approval sought for the following year's plans and capital funding requirements.

The capital budget submission provided detailed financial data on each capital project including the estimated impact on future operating budgets (i.e. additional staff to support and maintain systems, elimination of positions, reduced overtime costs, etc.).

An examination of the Service's capital budget submissions, which appear to have been in accordance with practice at the time, indicated that efficiency savings and productivity gains were included in the Impact on the Operating Budget section of the capital budget submission even though these efficiencies were not necessarily expected to translate into actual budget reductions.

The initial approval of the Metropolis project was done on the basis of anticipated efficiency savings and improvements to the effectiveness of policing in the City. However, with the changes in the composition of the Toronto Police Services Board and Metro Council since the initial approval of the project, the inclusion of these efficiency savings in the operating budget impact may have led Metro Council and Toronto Police Services Board members to interpret this as opportunities for actual budget reductions, especially if read in isolation from other information presented (i.e. benefits studies, presentations by Metropolis project team, Metropolis status reports, etc.).

Recommendations:

It is recommended that:

5.The City of Toronto, Budget Division, clarify to all Departments, Agencies, Boards and Commissions that the Impact on the Operating Budget section of capital budget submissions be restricted to actual budget changes; and

6.Future submissions to the Toronto Police Services Board and Council clearly identify the types of benefits to be realized (i.e. actual dollars vs efficiency savings).

Metropolis Project Costs Records

The Services' current financial accounting system does not have the capability to provide for detailed monitoring of budgeted costs by project. In order to adequately monitor the project, detailed manual records had to be maintained by C&T. Our sample review of C&T's manual records indicated that individual project cost information was well documented and properly supported. While this method of monitoring the project costs worked in this instance such "shadow" accounting systems generally represent an inefficient method of achieving their goal. The Service would be better served by an accounting system which tracked project costs for the various Units of the Service.

Recommendation:

It is recommended that:

7.Police Service staff investigate the possibility of incorporating project accounting capabilities into any future financial systems.

Control of Individual Project Costs/Budget

Metropolis was approved as one large project by Metro Council and although it was made up of over 100 sub projects there was no requirement to report on each individual project. The $60 million project budget was monitored on a global basis. Each year, the remaining funds available for the Metropolis project were reviewed and allocated to the various individual projects as prioritized by the Priorities Review Committee, consisting of the Chief, Deputy Chiefs, Director of IT and the CAO, Policing.

Individual budgets expressed in dollars and person hours were developed for certain projects. However, there was a general lack of control in determining whether individual projects were on budget.

Systems development was performed utilizing a mix of internal staff and consultants. Salaries of internal staff were treated as a fixed cost and were not charged directly to individual projects. As a result, the cost of a project was largely dependent upon whether internal staff or consultants were used. Since internal staff time was not tracked there was no mechanism to monitor and control the true costs of individual projects.

In a report to the Chief of Police dated January 28, 1992, the former Metropolitan Auditor made a number of suggestions with respect to the administration and control of the Metropolis project. Included was a recommendation that the budgetary approval process for system development provide for full costing and that project costs should include internal as well as external costs. Although the original Metropolis submission included internal costs, these were removed from subsequent submissions. Because internal staff costs were not captured, the total cost of the projects are not accurately reflected.

In our analysis of project costs and benefits, it was noted that 11 projects accounting for 36 percent of the total efficiency savings were funded out of the operating budget using internal staff and assigned zero cost. Nearly all of the actual dollar savings (90%) was attributed to the CASC Court Scheduling system, which was also funded out of the operating budget and assigned a zero cost.

To truly compare the costs of Metropolis to the benefits received, adjustments must be made to either the costs or the benefits. If the benefits were adjusted, the table on page 3 would show a reduction in the total annual benefits in excess of $20 million. Thus the ongoing annual benefits would be $27.1 million rather than the stated $47.6 million. Such an adjustment would also reduce the cumulative savings identified in the table above by approximately $54 million for a revised total cumulative savings of about $85.8 million. Alternatively, the internal costs could be added to the $60 million stated cost of Metropolis for comparison against the benefits listed on page 3. An internal report prepared by the Service estimates that contributions from internal resources towards systems development is approximately $15 million. This however, is a very rough estimate since there were no accurate records indicating staff time per project.

Recommendation:

It is recommended that:

8.Any cost/benefit analysis performed should include both external as well as internal costs.

Conclusion:

Prior to the Metropolis project, the Toronto Police Service's information technology environment consisted mainly of mainframe computers and terminals with limited applications supporting policing functions. Metropolis expenditures were largely comprised of a city-wide data communications network, server hardware, desk top computers and third party commercial software.

These hardware and software purchases accounted for approximately 65 percent ($40 million) of total expenditures. The remaining $20 million were comprised of $9 million for infrastructure items such as training, maintenance/support, security and disaster recovery and $11 million for application development and consulting.

With the exception of the claimed $3.25 million reduction in court overtime expenses, the $60 million Metropolis investment had limited impact on the operating budget in terms of actual dollar savings and/or budget reductions. It should, however, be noted that the Metropolis project was initially approved on the basis of time savings and productivity gains. The fact that these efficiency savings were quantified and expressed in monetary terms may have confused the issue and may have led Metro Council and Toronto Police Services Board members in believing that the project would result in potential budget reductions. This is more of a problem in how such savings are quantified and reported rather than a problem with the Metropolis project itself.

Metropolis initiatives have helped streamline various labour intensive processes resulting in efficiency savings and productivity gains. The value of these savings, however, is dependent upon how resources have been redeployed. Because appropriate, relevant and measurable benchmarks were not established at the beginning of the project to track resource redeployments, the valuation of the efficiency benefits as provided by the Toronto Police Service could not be substantiated.

From a technology standpoint, the project successfully upgraded the Service's outdated technology. In addition, applications such as case processing, mugshot and finger printing systems developed to improve policing functions, while not necessarily resulting in budget reductions, nevertheless have value with respect to the effectiveness and credibility of the Police Service.

Contact Name:

Jerry Shaubel, Director of Audits, 392-8462"

The Board was also in receipt of the following report MARCH 10, 1999 from David J. Boothby, Chief of Police:

"SUBJECT:MANAGEMENT RESPONSE - CITY AUDITOR'S REVIEW OF THE METROPOLIS PROJECT

RECOMMENDATION:THAT the Board receive this report for information purposes.

BACKGROUND:

The Police Services Board, at its meeting to be held March 26, 1999, will consider a report dated March 1, 1999 from the City Auditor relative to a review of the Service's Metropolis project.

The $60M Metropolis investment has been a critical and essential component over the last several years in terms of the Service's ability to sustain and improve services to its Public, while reducing overall operating costs and staffing levels. The benefits to the Police and the City have been significant, and include examples such as the Alternate Response Unit process, streamlined criminal information processing, enhanced and streamlined forensic identification services, efficient communication systems, and many, many others. Without this investment, the Service would not have achieved current service levels in its delivery of community policing services. It is essential that the Service sustain this investment in a state of good repair in order to continue its service delivery at current performance levels. It is in this context that the Command Officers have reviewed the City Auditor's report, and the Command supports the forward-looking nature of the recommendations. My comments on each of the Auditor's eight (8) recommendations are as follows:

Recommendation 1

Project benefits calculations / projections be reviewed by Police Services management for validity and reasonableness prior to incorporation into any report submitted to the Board and / or Council;

The Service agrees. The Service recognizes the difficulty of estimating benefits - in that over time the actual benefits achieved may be influenced by the continual changing of both internal and external factors. In the specific cases referenced, the Service does view the benefit estimates as reasonable, however, validating this would require extensive and complex efforts, and may still be subjective. For future projects the Service will more clearly document the rationale for all benefit estimates, and ensure additional scrutiny of same by senior management.

Recommendation 2

Where projects and related benefits span several years, the cost savings recognized in prior years be reviewed annually to ensure that such benefits may still appropriately be included in any cumulative annualized benefits calculations;

The Service agrees and will be so guided in future. The Service does believe, however, that the overall benefit estimates remain reasonable.

Recommendation 3

Clear parameters be established at the beginning of each project, identifying its scope and the benchmarks to be used in evaluating the success of the project. Such benchmarks must be relevant, objective, and measurable;

The Service agrees. Some projects (as is the case with many components of METROPOLIS) do require considerable management judgement in accepting whether efficiency estimates are reasonable. For example, performing extensive time and motion studies before and after the implementation of a Service wide electronic mail system may not produce any more reliable estimates than management intuition. Where appropriate, the Service will establish measurable benchmarks, along with a process to manage benefits. For instance, in the Occurrence Reengineering project, the plan for achieving labour savings is very specific and measurable; with projects such as "Y2K" however, the benefits (benchmarks) are accepted as intuitive (operational systems in time for 2000).

Recommendation 4

Where projects are approved on the basis of efficiency savings, a plan should be established outlining how staff redeployments will be achieved, and a reporting process be developed to identify such redeployments so that they may be properly reflected in future operating budgets. In the event that the efficiency savings will be absorbed by increases in service levels or demands, these increases should be clearly identified;

The Service agrees to this in principle and will adhere to City guidelines. Note that in both the public and private sector it is commonplace to cope with major work force reductions through Executive Management strategies such as restructuring, reengineering, and retooling. The Service has employed each of these in its organizational changes during the past 10 years.

Recommendation 5

The City of Toronto, Budget Division, clarify to all Departments, Agencies, Boards and Commissions that the Impact on the Operating Budget section of capital budget submissions be restricted to actual budget changes;

The Service will adhere to the Budget Division guidelines for capital budget submissions.

Recommendation 6

Future submissions to the Toronto Police Services Board and Council clearly identify the types of benefits to be realized (i.e. actual dollars vs efficiency savings);

This has been the practice in the Service for METROPOLIS reports and other project reporting and will continue to be so.

Recommendation 7

Police Service staff investigate the possibility of incorporating project accounting capabilities into any future financial systems;

The Service has already investigated this item and has identified project accounting as a mandatory requirement. It is our understanding that the SAP system acquired by the City can meet this requirement.

Recommendation 8

Any cost / benefit analysis performed in support of projects should include both external as well as internal costs.

The Service agrees and will comply with guidelines provided by the City. The internal process for approving and prioritizing projects (the Project Review Committee process) takes into consideration internal operating (staffing) costs in addition to any Capital (external / consulting) costs. It is the Service's understanding that our budget submissions have complied with City guidelines in effect at that time.

Messrs. Larry Stinson, Director, Information Technology Services (8-7550) and Frank Chen, Director, Finance and Administration (8-7877) will be present at the Board meeting on March 26, 1999 to respond to any questions regarding the foregoing."

Conclusions:

The Board received the foregoing and approved the following Motion:

THAT the Board send copies of the foregoing reports to the City of Toronto Audit Committee for information along with a request that they be forwarded to City Council for information.

Contact Name and Telephone Number:

Larry Stinson, Director, Information Technology Services, telephone 808-7550.

Respectfully submitted,

Norman Gardner

Chairman

A:\mtpolaud.doc