Evaluation
State Evaluators
Pacific Institute for Research and Evaluation
The primary components to the State-Level SPF SIG Evaluation are the Community Level Instrument (CLI), the Community Survey, and local process data collected through the PIMS. In addition, interviews will be conducted with key stakeholders at the state and local level to assess the process of implementing the SPF SIG. Finally, and perhaps most importantly, data collected at the state level on the outcome measures of interest will be collected and analyzed.
Local level evaluators in New Mexico are charged with evaluating the effects of the SPF SIG at the local level. This should include both process measures and outcome measures. Local improvements in infrastructure and capacity that are a direct result of the SPF SIG should be recorded, as well as changes in local contributing factors and if possible consumption patterns and outcomes. Finally, the local evaluation should also consider changes in cultural competency that are attributable to the SPF SIG, and the sustainability of prevention efforts once the grant expires.
National Cross-Site Evaluation: Community Level Instrument (CLI)
The CLI is an on-line reporting tool that collects detailed information about each strategy a program implements. Each community representative is provided a login User ID and password from Westat, the CLI contractor. Typically, the SPF SIG program coordinator is the person responsible for completing the CLI, although it is assumed they will need input from the local evaluator and other staff persons. Some communities ask that the local evaluator be responsible for completing the CLI. Regardless of how it’s delegated, while only one person has access to the data reporting system, many will contribute. It is strongly recommended that programs print the survey and answer the questions on the paper format first before transferring responses to the web-based version. Once a section is submitted, it is almost impossible to go back and revise responses.
The web address for accessing the CLI is: https://cliwsmsprd.westat.com/ The individual delegated by each program will be sent a User ID and password by email.
After the data are cleaned, each community is able to access the data they enter. The State Evaluator is able to access data from all the programs in the state. Additionally, there is a data analysis tool on the CLI website that will allow you to conduct some basic frequencies and cross-tabulations on your data.
If you have questions about the CLI, you should first contact Martha Waller at mwaller@pire.org. If Martha is unable to answer your question, you can also contact cli@mayatech.com for technical assistance.
Revisions have been made to the CLI since February 2008. To read about those revisions please read: Changes to the CLI (provide link to Changes to the CLI document)
Before completing the CLI, please read: Guidelines for Defining an Intervention (provide link to Defining and Intervention document)
CLI documents
NM SPF SIG Community Survey
NM Community Survey FY 09
The Community Survey is designed to assess changes over time in the contributing factors related to the targeted intervening variables. Data are collected in the targeted SPF SIG communities and in non-targeted comparison communities.
During FY 07, a statewide phone survey was conducted using random-digit-dialing (RDD) to collect information about attitudes, knowledge, and behaviors of persons 18 to 25 years old in NM. Unfortunately, a very low response rate to the phone survey meant that the data were not sufficient to draw conclusions.
During FY 08, the Community Survey was revised to address formatting and content concerns and a new recruitment approach was taken to increase the response rate and reduce burden on the local providers and evaluators. With some exceptions those programs with local Motor Vehicles Division (MVD) offices solicited assistance from the local MVD supervisor and staff to recruit clients for the survey. Staff were trained on what to say about the survey and were provided with incentives to motivate them to recruit. MVD staff were to present a recruitment card (written in both English and Spanish) to every client 18 years old or over, explain the survey, ask if the client would like to participate, and have the individual circle yes or no on the card. If the client agreed, the participant was also asked to provide a first name only, a contact email or phone #, and the town or community of residence. The card was then collected by the MVD staff person and put into a box. The client was given a small incentive for completing the card. These cards (both yes and no cards) were to be collected weekly and mailed to the state evaluator. Those who responded yes and provided contact information were either contacted by email or by phone. If contacted by email, respondents were sent to a website where they could enter their email addresses each with a unique password provided, in the email invitation, and then they could complete the survey. Those who chose to be contacted by phone were interviewed in the telephone by a PIRE employee. Respondents could complete the web or phone interview in English or in Spanish. All those who completed the survey were entered into a drawing for 24 prizes worth $150 (e.g., a $150 gas card or Ipod) and were contacted using the email or phone # provided.
Some communities where MVD recruitment was not feasible, recruited at local events, stores, and other locations using the entry cards described above. Communities with mainly Native American populations had respondents complete a paper and pencil version of the survey rather than use the recruitment cards. This direct method allowed them to address any questions and concerns about the survey immediately and provided access to people who may not use internet or phone service frequently. Finally, UNM requested from their registrar a list of 2000 randomly selected undergraduates and their email addresses. Invitational emails were sent directly to those students rather than using recruitment cards.
All data recruitment methods received approval from PIRE’s Institutional Review Board (IRB) and UNM’s recruitment additional received approval from their IRB.
For FY 09, communities will conduct paper and pencil surveys in strategic locations in their communities in addition to recruiting for the internet survey using pre-printed recruitment cards with the web address for the survey pre-printed on the cards. Each community will write a community specific data collection protocol specifying the target groups that will be sampled, the people who will be involved in the data collection process, the training that will be provided to them, and the times and locations where recruitment will take place as well as the type of data collection or recruitment. Documents for the FY 09 Community Survey can be found under NM SPF SIG Project Documents.
New Mexico Prevention Information Management System (PIMS)
Most of you are intimately familiar with the PIMS data reporting system (www.nmpims.com) and entering data into the system. As of FY 09 some changes have been made to the SPF SIG reporting system on PIMS to align it more closely to the data required on the CLI. This was done for several reasons. When local programs first completed the CLI in February 2008, many questions about the strategies came as a big surprise to programs. It was difficult to find that information after such a long time period and programs struggled with completing the CLI. It is encouraged that programs use the PIMS system in order to conveniently CLI data every 6 months.
The PIMS for direct services has seen a face-lift, based on feedback provided by the preventionists using the system. We hope the improvements will help prevention programs not only with data entry, but also with data retrieval.
Qualitative versus Quantitative Data Collection for Evaluation
Quantitative data are very helpful in an evaluation and therefore much focus is placed on how to measure quantifiable changes over time. However, qualitative data are equally important and can provide great insight into how changes occur and the processes through which this changes took place. Since the SPF model is concerned with sustainability of efforts after the completion of the grant, knowledge of how programs worked to implement the SPF SIG and the challenges faced, processes that were created to address change implement the grant, etc. are all important in understanding the context in which a grant is taking place and can provide great insight about the successes or failures. Therefore, the combination of quantitative and qualitative data will be used to contribute to the state level evaluation.
|
