Assistive Technology Quick Reference Series

 

 

AT Outcomes Measurement

 

 

 

 


Accountability and cost-benefit analysis have become common topics of discussion throughout business and industry. The importance of understanding how systems perform and operate, including how well they meet their objectives and goals, is an integral part of effective management.   Rehabilitation services and education programs have been working to develop better performance indicators and strategies to deliver services that help individuals reach specific goals.  One aspect of this emerging focus is on the use and impact that assistive technology devices or services have on the outcomes of rehabilitation and education services.  Assistive technology, or "AT", is an important component of these programs and includes a wide range of aids, devices and strategies that are used to improve, enhance, or maintain an individual's ability to perform certain tasks and activities.   Determining how well the AT works, and more importantly how well it enhances service outcomes, is basically what "AT Outcomes" is all about.

 

"Assistive technology outcomes" refers to what happens as a result of using an AT device or service.  The "outcome" from the use of "AT" can cover a wide range of issues and goals, and could include whether:

  • Technology was used or abandoned;
  • The AT led to efficient completion of specific tasks/activities;
  • The AT played a role in gainful employment;
  • Cost savings were realized;
  • Decreased family or caregiver support was realized;
  • Increased independence resulted; or
  • The individual’s quality of life was enhanced.

 

This quick reference guide provides an introduction to many of the issues and challenges faced in measuring AT Outcomes in rehabilitation programs.

 

PURPOSE OF AT OUTCOMES MEASUREMENT

 

Why would rehabilitation programs want to measure AT outcomes?  The specific purposes are going to differ according to individual stakeholder perspectives, but there are a number of points that have relatively universal appeal and meaning.  Experiences of a single individual, while important, may only reflect unique circumstances and personal preferences and may not give an accurate appraisal of the benefit of using AT. However, when outcomes data is captured across many people, trends become apparent and can be used to:

  • Identify what AT devices work for whom and why -- and use this information to guide future decisions by all stakeholders;
  • Measure an individual's progress toward a vocational or educational goal;
  • Justify continued services and equipment for an individual AT user;
  • Identify performance areas in which a consumer may have unmet needs;
  • Measure the efficiency and effectiveness of AT services; and
  • Justify future AT services and equipment based on a history of successful (and unsuccessful) cases. 

 

Although the practice of vocational rehabilitation differs somewhat between public and private rehabilitation, the focus is clearly on employment.  The contribution of assistive technology in reaching, and maintaining, successful employment outcome, is also similar.  Typical goals for using AT could include how well AT helped to:

  • Increase the effectiveness and efficiency of task performance by the end user;
  • Increase end user's employability;
  • Increase the end user's productivity once employed; or
  • Increase the end-user's participation in community living activities.

 

Influence of various reimbursement models

There are three models of practice and reimbursement under which most AT is recommended.

Each has distinct priorities.

·         The medical model emphasizes services and equipment that are medically necessary;

·         The educational model emphasizes achievement of educational goals;

·         The vocational rehabilitation model emphasizes achievement of vocational goals.

 

In practice, these distinctions between medical, educational, and vocational priorities have a tremendous influence on the types of services and equipment that are funded. Generally speaking, none of the three models provide reimbursement for follow-up studies by AT service providers, nor do they require systematic capture or outcomes data.

 

FREQUENTLY ASKED QUESTIONS

 

Who are the stakeholders and how would they benefit from improved AT outcomes measurement? 

There are a number of stakeholders who would benefit from the results of better AT outcomes measurement.

·         Consumers can establish reasonable expectations about the relative advantages and disadvantages of specific AT devices.

·         Counselors can make more informed decisions about funding AT services and equipment.

·         VR administrators can make better programmatic and budget decisions about the impact of AT.

·         AT Practitioners can make more informed recommendations based on past experiences.

·         Employers can become aware of costs and productivity benefits associated with AT use.

 

What are examples of AT outcome measures? 

Examples of AT outcome measures are often specific to the environment and social context of use.  Specific to vocational rehabilitation applications, there are a number of examples that are relevant to most stakeholders:

  • Achieving vocational goals as a result of using AT;
  • Improving skills and performance areas that affect employability;
  • Improving skills and performance areas that affect productivity;
  • Expanding vocational options as a result of using AT or accommodations;
  • User satisfaction with AT devices or services.

 

Is there reliable information available about device abandonment? 

The information in the literature on abandonment of AT devices is very limited.  Questions exist on the reliability of many estimates since device "abandonment" may actually be a positive indicator of functional gains made by an end-user.  For example, an individual given AT as part of their hospital rehabilitation program may cease needing a walker once strength and balance improve.  More thorough follow-up is needed across varied settings to have a better understanding of how AT devices continue to be used.

 

Why aren't AT outcomes typically measured? 

The age-old reasons: time and money!  It's simply not part of the daily routine for VR counselors, case managers and AT practitioners.  From the VR perspective, it's especially challenging because agencies already collect a large amount of data on their consumers, but unfortunately this data is often not specific enough to tease out the effects of AT.  As a result, most rehabilitation programs and agencies do not have information about the cost-effectiveness of AT. 

 

For their part, AT practitioners have usually not been required to provide follow-up information in order to document the effectiveness of their services and device recommendations, so they have not allocated time to complete this process.

 

Are there other reasons that AT outcomes haven't been better measured? 

As in other areas of healthcare, it is incumbent upon the practitioner to provide evidence that their recommendations will be effective -- assuming that they have been asked to provide this information.  To date, this hasn't happened for several reasons. 

  • The medical model payers (Medicare, Medicaid, HMOs, private insurance) have imposed funding and reimbursement criteria for AT that are based on cost containment, not device effectiveness and efficiency, so they do not base initial reimbursement decisions on outcomes data.
  • In vocational rehabilitation and educational models progress is typically reviewed once per year, often without the AT practitioner being included in the assessment of progress.  There are very few programs that we are aware of that incorporate any assessment tool that specifically collects information about the status of AT products or services.

 

Thus, AT outcomes measurement has not happened historically because the respective systems have not been designed to facilitate or require that it happen.

 

 

PREPARING FOR MEASURING AT OUTCOMES

 

Quality improvement efforts such as AT outcome measurement should be important priorities for rehabilitation agencies and their professional staff.  The time and effort needed to plan, implement and monitor the contribution of assistive technology resources and services, while not being extensive, does represent a clear commitment to ensuring that programs and agencies are using their resources in the most effective manner possible to achieve the best employment outcomes for the persons being served.  Current trends reinforce the necessity of capturing better AT outcomes information.  Programs and agencies are serving persons with more severe disabilities; providing services from new venues such as "one-stop" centers; partnering with other programs and agencies; and finding funding sources demanding better accountability for expenditures and service outcomes.

 

How can counselors help? 

Counselors will realize immediate benefit from their contribution to tracking AT outcomes in a couple of ways.  Technology-related decisions with current consumers should be aided from the better awareness for AT overall, plus the ongoing attention given to AT will help to ensure that technology options are given adequate consideration.  Counselors will contribute a great deal to improving the quality of future decisions to support or deny requests for AT assessment services and subsequent recommendations for AT devices by doing the following:

  • Specify consumer vocational and educational goals as clearly and specifically as possible on AT referrals;
  • Require that AT practitioners justify AT devices in terms of the individual's vocational goals and expectations;
  • Require AT practitioners to describe the relative advantages of AT devices, including objective measures of the consumer's performance with and without the AT device
  • Allocate "billable hours" for AT practitioners to follow-up on their recommendations to identify what worked, what didn’t work, and why.

 

Suggestions for VR programs interested in collecting AT outcomes information

Efforts to look at the role of AT in any rehabilitation program should be completed as part of other quality assurance initiatives.  Involvement from key administration and management staff will be needed to develop effective strategies that will capture the necessary information in the most efficient manner possible.  Counselors and case managers often feel over-burdened with existing data collection requirements.  Before decisions are made for counselors to collect additional data, agencies should look at their overall approach to outcomes measurement and obtain direct input and suggestions from AT providers.

 

Here are some suggestions on data to gather and what to look for:

·         Keep counts of the services and technology that are provided.  When combined with follow-up information from consumers and providers, this can form the basis for an outcomes study;

·         Look at the effectiveness of AT toward facilitating intermediate tasks and role demands that might lead to an individual's employment;

·         Examine the effectiveness of AT at key stages of the employment process: vocational evaluation, job exploration, training, placement, and follow-up;

·         Measure the achievement of short-term goals that indicate progress toward employability;

·         Identify employment outcomes that have been affected by AT use; and

·         Measure the value and efficiency of AT-related services that counselors are authorizing in terms of:

    • Timeliness of AT services;
    • Timeliness of reports;
    • Quality of information in the reports;
    • Opportunities for consumer trial of AT devices during the evaluation period;
    • Relevance of the recommended AT devices to the consumer's goals and life circumstances.

 

 

 

OTHER INFORMATION RESOURCES

 

The Federal Government is sponsoring several research activities investigating the area of AT Outcomes.  Three centers have been funded for the purpose of identifying best practices in outcomes measurement. 

 

CATOR -- Consortium of AT Outcomes Researchers

Duke University

DUMC 3888, Durham, NC  27710

Phone: 919-684-6271; Fax: 919-681-9984

http://www.AToutcomes.org

 

ATOMS -- AT Outcomes Measurement System

Occupational Therapy Program, University of Wisconsin-Milwaukee

P.O. Box 413, Milwaukee, WI  53201-0413

Phone: 414-229-6568; Fax: 414-906-3959

TTY: 414-229-5628

E-mail: atoms@uwm.edu

http://www.atoms.uwm.edu

 

NATRI -- National Assistive Technology Research Institute

Department of Special Education and Rehabilitation Counseling

229 Taylor Education Building, University of Kentucky

Lexington, KY 40506-0001

Phone: 859-257-4713

E-mail: natri@coe.uky.edu

 

Web-Based Resources:

 

· AT Outcomes Resource Site

This is a good site that is maintained by the Adaptive Technology Resource Centre for general information on AT Outcomes, information on a number of existing outcome measurement tools and a discussion listserv to stay up on developments around AT outcomes.

http://www.utoronto.ca/atrc/reference/atoutcomes/

 

Useful Articles:

 

·         DeRuyter, F. (1995). Evaluating outcomes in assistive technology: Do we understand the commitment? Assistive Technology, 7, 3-8.

·         DeRuyter, F. (1997). The importance of outcome measures for assistive technology service delivery systems. Technology and Disability, 6, 89-104.

·         Fuhrer, M. J. (2001). Assistive technology outcomes research: Challenges met and yet unmet. American Journal of Physical Medicine and Rehabilitation, 80, 528-535.

·         Minkel, J. L. (1996). Assistive technology and outcome measurement: Where do we begin? Technology and Disability, 5, 285-288.

·         Smith, R. O. (1996). Measuring the outcomes of assistive technology: Challenge and innovation. Assistive Technology, 8, 71-81.

·         Warren, C. G. (1993). Cost effectiveness and efficiency in assistive technology service delivery. Assistive Technology, 5, 61-73.

 

 

 


This document was written and developed by Jim Lenker and Tony Langton, with additional contributions from Joy Kniskern, Frank Coombs, Hunter Ramseur, Karen Milchus, Christie Ramseur, Wil Morales, and Stephen Sprigle.