2002-03 Balanced Scorecard Metrics
The Balanced Scorecard is a management technique designed to provide a view of an organization from four perspectives: user, internal processes, financial, and future/learning potential. This document explains how the University of Virginia Library is defining each of these four areas by providing the details of the metrics we are using in determining our success in each perspective.
Each of the metrics described below contains two targets. Target1 indicates complete success in achieving the metric; Target2 indicates partial success. Management Information Services will compile and report the results of all metrics on a regular basis.
USER PERSPECTIVE: How well is the library meeting the needs of our users?
- PROVIDE EXCELLENT SERVICE TO USERS OF THE UNIVERSITY OF VIRGINIA LIBRARY
METRIC I. OVERALL RATING IN STUDENT AND FACULTY SURVEYS.
Target1: We should attain a score of at least 4 (out of 5) from each of the major constituencies: undergraduates, graduates, humanities faculty, social science faculty, science faculty.
Target2: Score of at least 3.9 from each of the major constituencies.
Method: The University Library conducts extensive surveys of our clientele approximately every other year. A final question in each survey asks the respondent to "rate your overall satisfaction..." with the Library using a 1 to 5 scale. This metric will consider the two latest surveys on record.
Separate Tallies: The ratings for faculty and students will be kept and reported separately.
METRIC II. CUSTOMER SERVICE RATING IN STUDENT AND FACULTY SURVEYS.
Target1: Customer services should receive a score of at least 4.25 (out of 5) from each of the major constituencies: undergraduates, graduates, humanities faculty, social science faculty, science faculty.
Target2: Score of at least 4.0 from each of the major constituencies.
Method: As part of its series of surveys, the University Library now asks respondents to rate three aspects of customer service: speed, courtesy, accuracy. The question refers very generally to customer service and thus could be applied to quality of service received at any service point: reference, circulation, e-centers, email, phone, correspondence, etc. Scoring is on a scale of one to five, five being "most satisfactory." The three scores will be combined into one composite number. This metric will consider the two latest surveys on record.
METRIC III. ARL INDEX RANKING.
Target1: The U.Va. Library should rank in the top 20.
Target2: The U.Va. Library should remain ranked in the top 25.
Method: The Association of Research Libraries (ARL) includes 112 of the largest academic libraries in North America. Each year ARL uses principal component analysis to aggregate five data categories: volumes held, volumes added (gross), current serials, total library expenditures, total professional and support staff. For each library this analysis produces the "ARL Library Membership Index."
- EDUCATE USERS IN THE SKILLS NECESSARY TO FULFILL THEIR INFORMATION NEEDS
METRIC IV. OVERALL INSTRUCTION RATING IN USER EDUCATION PARTICIPANT SURVEY.
Target1: An overall rating of 4 on a scale of 1 to 5.
Target2: An overall rating of at least 3.8.
Method: Evaluation will ascertain how well the training received correlates to the needs of the audience. A brief survey of 3-4 questions developed by Management Information Services (MIS) and the User Education Committee will be distributed via e-mail to a sample of participants of course-related user education classes at the end of the semester during which instruction is received. The survey will include an overall rating question.
[For 2002/03, we will continue to survey classes where there is easy access to student e-mail addresses, but will suspend surveying of classes where data collection difficulties exist.]
METRIC V. ATTENDANCE AT SPECIAL COLLECTIONS SESSIONS.
Target1: Increase by 20% annually the number of informational, instructional, and outreach sessions and the number of persons attending sessions in Special Collections.
Target2: Increase by 10% the number of sessions and the number attending.
Method: Special Collections will count and record the number of sessions and persons attending each fiscal year.
- SELECT, BUILD, MAINTAIN, AND PRESERVE HIGH QUALITY COLLECTIONS THAT REFLECT THE NEEDS OF THE LIBRARYÃŒS USERS AND SUPPORT THE UNIVERSITYÃŒS MISSION
METRIC VI. CIRCULATION OF NEW MONOGRAPHS.
Target1: 60% of all newly cataloged monographs (excluding reference books and other non-circulating materials) should circulate to U.Va. patrons within two years of the time of cataloging. Usage of newly acquired or newly digitized electronic monographs will be measured separately.
Target2: 50% of all new titles should circulate within the first two years. Usage of newly acquired or newly digitized electronic monographs will be measured separately.
Method for print material: A program will extract data from the SIRSI records, enabling us to examine the two-year circulation of print monographs.
Method for digital material: The procedures for tallying use of digital monographs are not yet established.
Separate Tallies: The numbers for print material and digital materials will be kept and reported separately.
[For 2002/03, we will suspend the effort to measure electronic monographs.]
- PROVIDE CONVENIENT AND TIMELY ACCESS TO THE LIBRARY'S COLLECTIONS
METRIC VII. TURNAROUND TIME FOR PROCESSING AND FILLING USER REQUESTS.
Target1: Satisfy the turnaround targets 90% of the time.
Target2: Satisfy the turnaround targets 80% of the time.
Searches: Notify patron of results of search within 4 days of request.
Recalls: Process requests within 1 day.
ILL: Fill requests in 7 days.
LEO: Fill requests in 7 days.
Toolkit: Fill requests by agreed upon date.
New Books: Fill requests in 7 days.
Method: Each category will require development of a separate methodology.
INTERNAL PROCESS PERSPECTIVE: How do the library's internal processes function to efficiently deliver library collections and services?
- ACQUIRE, CREATE, ORGANIZE, PRESERVE AND DELIVER INFORMATION RESOURCES IN A TIMELY, EFFICIENT, AND ACCURATE MANNER
METRIC VIII. PROCESSING TIME FOR ROUTINE ACQUISITIONS.
Target1: 90% of in-print books from North America will be processed within one month.
Target2: 80% of in-print books from North America will be processed within one month.
Method: This metric will estimate the elapsed time from ordering to delivery at the destination library. Certain classes of materials will be excluded: user requests, non-book materials, and orders for Special Collections, Law, and Health Sciences. The methodology will rely on SIRSI records for Yankee and New York Times/Washington Post approval orders.
METRIC IX. TURNAROUND TIME FOR SHELVING BOOKS.
Target1: Accurately shelve/refile 85% of all materials by the end of the day following discharge regardless of where the item was discharged.
Target2: Accurately shelve/refile75% of all materials by the end of the day following discharge regardless of where the item was discharged.
Method: Once a week, a sample of 20 items from all libraries will be selected from the uses and discharges of the previous day for checking in the stacks. The overall annual percentage is determined by dividing the number of items meeting the criteria by the total sample size.
- USE RESOURCES IN THE MOST INNOVATIVE, EFFICIENT, AND EFFECTIVE WAY POSSIBLE
METRIC X. STAFF SURVEY RATING OF INTERNAL SERVICE ACTIVITIES.
Target1: Composite rating of 4.0 of all internal service activities in annual staff survey, with no units rated below 3.5.
Target2: Composite rating of 3.5, with no units below 3.0.
Method: The University Library will conduct a staff survey once a year. As part of the survey, each staff member will have the opportunity to rate the performance of each internal service unit or activity (e.g., Administration, Acquisitions, Cataloging, Budget, Business Services, Communications and Publications, Interlibrary Services deliveries, Library Human Resources, Library Information Technology Services, Management Information Services) on a 1 to 5 scale.
- DEVELOP WITHIN THE LIBRARY A CULTURE OF ASSESSMENT
METRIC XI. PERCENTAGE OF DEPARTMENTS WITH AN ASSESSMENT PLAN.
Target1: 100% participation.
Target2: 80% participation.
Method: Our goal should be to increase the quality and quantity of assessment activities as we move toward a culture of assessment. To further this goal, each department or other unit that reports to the University Librarian or an Associate University Librarian should develop an assessment plan to coordinate assessment activities within their unit. MIS will assist unit managers as they develop plans and will monitor and tally the results for this metric.
[For 2002/03, suspend this measurement, but departments that find assessment planning helpful are encouraged to continue
- CONTINUOUSLY IDENTIFY, REVIEW, AND IMPROVE THE PROCESSES THAT HAVE THE GREATEST IMPACT
METRIC XII. PROCESS IMPROVEMENT.
Target1: Complete at least one major process improvement each year.
Target2: Complete at least one major process improvement every two years.
Method: Each year the Library seeks to identify major activities, review relevant policies and procedures, and make changes that will improve efficiency. By July 1 the Associate University Librarian for Organizational Development will send a message to all library staff asking them to submit ideas for library processes that could be improved using new technology or more efficient methods. The Administrative Group will consider the proposals and decide on the process improvement project for the year. The AUL for Organizational Development will monitor this metric. (Examples: The shelving and 3-C timeliness initiatives both improved library services.)
FINANCE PERSPECTIVE: How well are the library's finances managed to achieve our mission?
- INCREASE THE FINANCIAL BASE THROUGH PRIVATE DONATIONS, OTHER EXTERNAL SUPPORT, AND INCREASED INSTITUTIONAL SUPPORT
METRIC XIII. NON-STATE EXPENDITURES AS A PROPORTION OF TOTAL UNIVERSITY LIBRARY EXPENDITURES.
Target1: The target for FY2002, our initial year for Balanced Scorecard, was 10%. The target will increase by one half of one percent each following year, i.e., the target for FY2003 will be 10.5%, for FY2004 it will be 11% etc.
Target2: The partial success indicator for each following year is .5% less than the Target1 indicator.
Method: Non-state expenditures include annual and unrestricted gifts, sponsored programs, grants, endowment and project income, and e-commerce. These expenditures will be compared with the total expenditures of the University Library. The Budget Director will work with the Director of Development to monitor this figure each year.
METRIC XIV. LIBRARY EXPENDITURES AS A PROPORTION OF THE UNIVERSITY EXPENDITURES.
Target1: The University Libraries should continue to account for at least 4% of the E&G (educational and general) share of the University expenditures.
Target2: The University Libraries should account for at least 3.75% of E&G.
Method: Total University E&G expenditures will be obtained from Institutional Assessment and Studies; Library expenditures will be obtained from Oracle. These two figures will be used to calculate the percent.
- PROVIDE RESOURCES AND SERVICES THAT HAVE A HIGH RATIO OF VALUE TO COST
Metric XV. EXPENDITURES FOR CIRCULATING MONOGRAPHS AS A PROPORTION OF ALL MONOGRAPHIC EXPENDITURES.
Target1: Monographs that circulate within the first two years should account for at least 60% of the monographic budget.
Target2: The circulated items should account for at least 50% of the monographic budget.
Method: A yearly study similar to the 1993-95 baseline study will be conducted. The study will track purchased monographs for two years to determine if a customer uses them.
METRIC XVI. UNIT COST OF SERIAL USE (PAPER AND ELECTRONIC)
Target1. 0% increase in unit cost each year.
Target2. Less than 5% increase in unit cost each year.
Method: Once each year we will survey use of a sample of 400 paper serials (both unbound current issues and bound back volumes) and of all electronic journals for which usage data are available. Data for bound volumes will be from SIRSI; data for e-journals from statistics regularly reported by vendors. The method for unbound paper serials has not yet been determined.
Separate Tallies: The numbers for print journals and electronic journals will be kept and reported separately.
[For 2002/03, suspend measurement for paper serials, but work with Collections Group, CCC, and other groups to ascertain if the data collection now in effect will be useful in determining which paper serials should be canceled.]
METRIC XVII. UNIT COST OF ONLINE DATABASES.
Target1: 0% increase in unit cost each year.
Target2: Less than 5% increase in unit cost each year.
Method: Divide UVa's annual payment for each database by the annual usage number for that database. A single measurement can be provided by dividing the total cost of all databases (with reported usage statistics) by total uses. Usage is defined as either searches of or accesses to the database, whichever figure is available from the vendor.
METRIC XVIII. UNIT COST OF PROCESSING LIBRARY MATERIALS.
Target1. Annual percent change in unit cost will be equal to or less than annual percent change in overall library operating expenditures.
Target2: Annual percent change in unit cost will be within 5% of the annual percent change in overall library operating expenditures.
Method: We will calculate the total expenditures and the number of transactions in each of three departments: Acquisitions, Cataloging, and Interlibrary Services. The unit cost is the expenditure divided by the number of transactions.
Separate Tallies: The unit costs for Acquisitions, Cataloging (including Preservation), and Interlibrary Services (including Instructional Scanning Services) will be tallied and reported separately.
METRIC XIX. UNIT COST OF PUBLIC SERVICE TRANSACTIONS
Target1. Annual percent change in unit cost will be equal to or less than annual percent change in overall library operating expenditures.
Target2: Annual change in unit cost will be within 5% of the annual percent change in overall library operating expenditures.
Method: Analogous to Metric XVIII above. We will calculate the expenditures and the number of transactions for each of three functions: circulation, instruction, and reference. The methods of data collection will vary for three functions. Reference and circulation will be based on a two week sample; for instruction we will use data for an entire year.
Separate Tallies: The unit costs for circulation, instruction, and reference will be tallied and reported separately.
[For 2002/03, suspend user instruction portion of this metric.]
LEARNING/FUTURE PERSPECTIVE: How well is the library positioned to ensure that goals are met in the future?
- FOSTER LEARNING AMONG ITS EMPLOYEES TO ENCOURAGE CREATIVITY, COOPERATION, AND INNOVATION
METRIC XX. DEVELOP INDIVIDUALIZED AND DEPARTMENTAL LEARNING PLANS.
Target1: 90% of library staff and departments have compiled learning plans.
Target2: 80% of library staff and departments have compiled learning plans.
Method: Develop individualized learning plans that outline both developmental and job-skills training needs for each library employee. In addition, develop departmental training plans for each unit that reports to the University Librarian or an Associate University Librarian. Percentages for staff and departments will be averaged together to arrive at the target percentage.
[For 2002/03, suspend metric; however, departments that find the process helpful are encouraged to continue.]
- RECRUIT, DEVELOP, AND RETAIN PRODUCTIVE, HIGHLY QUALIFIED STAFF
METRIC XXI. RETENTION RATE OF COMMENDED EMPLOYEES.
Target1: 95% of commended employees remain on the staff.
Target2: 90% of commended employees remain on the staff.
Method: Staff members who have received at least one extraordinary contributor write-up and faculty who have received at least one "extra-meritorious" or "highest" salary increase over the past three performance cycles are identified at the beginning of each fiscal year. The Library may also take other forms of recognition into consideration. At the end of the year, we will determine what percentage of these employees still work at the library.
- PROVIDE FACILITIES THAT PROMOTE STAFF PRODUCTIVITY, ENCOURAGE LIBRARY USE AND ENSURE TOP QUALITY SERVICES
METRIC XXII. RELIABILITY OF EQUIPMENT.
Target1: Staff and user workstations (computers, scanners, video, audio, microform, copy equipment, etc.) shall function properly at least 95% of the time.
Target2: Equipment shall function properly at least 90% of the time.
Method: A random sampling of 20 workstations will be surveyed every other week. A workstation is considered down if any core function of the workstation is unusable when checked (e.g. if the only printer for a computer workstation is inoperative, the workstation is considered down, despite being able to perform other functions).
METRIC XXIII. RENOVATION OF MAJOR STAFF AND PUBLIC AREAS.
Target1: Three or more major renovations per year of which at least one is a staff area and one is a public area.
Target2: Two major renovations per year of either staff or public areas.
Method: Renovation of existing space allows the Library the opportunity to improve service and increase staff satisfaction and productivity. The Director of Business Services will solicit suggestions for renovations at the same time as budget requests are made. Administrative Group will decide which areas will be renovated.
- DEVELOP AN INNOVATIVE AND EFFECTIVE INFORMATION TECHNOLOGY INFRASTRUCTURE TO SUPPORT LIBRARY SERVICES AND VARIOUS INITIATIVES
METRIC XXIV. EXPENDITURES FOR DIGITAL MATERIALS.
Target1: U.Va. should be in the top quarter of all ARL libraries in our percentage of collections dollars expended on digital materials.
Target2: U.Va. should be in the top third of all ARL libraries in our percentage of collections dollars expended on digital materials.
Method: MIS will determine, and report, the percentage of the collections budget spent on digital materials at the end of the fiscal year.
METRIC XXV. USABILITY TESTING OF LIBRARY WEBSITES.
Target1: At least 75% of all new and revised library websites receive heuristic evaluation and at least 25% undergo full or discount usability testing.
Target2: At least 50% of all new and revised library websites receive heuristic evaluation and at least 25% undergo full or discount usability testing for the first year.
Method: The Library's Web presence provides the key entry point for our users, making it the hub of our technology infrastructure. Ease of use is a key ingredient to building a customer-orientated system. The web usability team will be responsible for assigning heuristic testing and arranging usability testing by library customers, both internal and external. This group will also tally and record the number of new websites receiving heuristic and usability testing and compare this number to the total number of new websites.
METRIC XXVI. INCREASING THE QUALITY OF DIGITAL ACCESS.
Target1: 100% of identified priority content will be produced per DCRT (Digital Contents Review Team) standards.
Target2: 90% of identified priority content will be produced per DCRT standards.
Method: Percentage completed is determined by comparing the number of projects DCRT identifies as priority content with the number of projects produced that meets standards as set by DCRT.