For this paper you will work as part of a team or individually to carry out an evaluation of a human service organization.  There are two options to choose from:

Option 1.   You can choose your current or former field placement or place of employment to conduct the evaluation.  Speak with your supervisor regarding an appropriate, feasible evaluation of interest to the agency.  Ask what data might be available for you to use.  Appropriate data include data regularly collected by the agency, or data that you collect yourself.  The data should not be aggregated, such as in a year-end report or annual report, but should be in a format that can be analyzed using Sofastats, SPSS, or some other statistical program.

Option 2.  You may use a hypothetical human agency, Human Services of Southern New Jersey, for the evaluation.  There is a website you may access containing information about two programs, the Girls Support Group and Substance Abuse Services.  There is also data that you may use for analysis.  The website is: http://humanservicesnj.weebly.com/ (Links to an external site.)Links to an external site.

Once there, you will be on the home page.  Then click on Family Support.  That page will give you a brief description of the Girls Support Group program.  For more information, click on Girls Support Group Reports.  Any information you do not find there you can create for purposes of this paper.

1. click on the link above, then click on the Family Support and then click Girls Support Group which is what our paper is about.

To access the Substance Abuse Services, click on Substance Abuse Services.  That page will give you a brief description of the services available for adolescents and adults.  Click on the Outpatient Adult Substance Abuse Treatment, then Reports.  You will find data that you can use for a paper.

Your report should no more than 10–14 pages long (APA style), not including charts. You should have 5–8 references, one of which should be Royse et al. Your paper should be submitted to the Dropbox and include a statement identifying the contribution of each group member to the writing of the paper. Only one member of the team needs to submit the paper.

For this evaluation report you are to use subheadings covering all the sections below:

1. Executive Summary (less than 1 page)
2. Background of the Program and Statement of the Evaluation Goals  (1 page)
   1. Describe the program, including its goals and objectives.
   2. List the goals of your evaluation.

• Brief Literature Review  (2–3 pages): What is known already about evaluating this type of program?

1. Methodology  (1–2 pages): How did you evaluate your program?
   1. Evaluation design
   2. Sample
   3. Data collection techniques
   4. Data collection instruments
   5. Data analysis
   6. Outcome measures
   7. Limitations of the study
2. Results  (1–2 pages
   1. Using descriptive statistics, describe the demographic characteristics of your sample, e.g., age, race, sex, and so forth.
   2. Using at least one or more statistical tests you think appropriate, describe the results of your data analysis to answer your evaluation question(s).
3. Discussion of Results  (2–3 pages)
   1. What do these findings mean? What could account for these findings other than the intervention?
   2. How do the limitations of the study (the design, instrument, sample, reliability, validity) influence findings?

• Conclusions and Recommendations  (2–4 pages)
  1. Conclusions: Based on your findings, what can you say about the program meeting its goals and objectives?
  2. Recommendations regarding the program: Based on your findings, what does the program need to do?
  3. Recommendations regarding subsequent evaluation of the program: What recommendations would you make to other researchers interested in evaluating this type of program? Be specific.

The grading rubric for the paper can be found in Course Essentials – Rubrics.

Video Instructions

The video below provides additional instructions, guidance, and tips for your Program-Level Evaluation Project.

FAQs Regarding the PLE Project

1. How do I pick an agency for the program evaluation?

   Ask your supervisor at your internship or place of employment whether the agency would like to have a program evaluation. Explain the assignment and the possible benefits to the agency, for example, a main benefit is to be able to demonstrate to funders and other constituents that the agency is trying to be accountable. In addition, a favorable finding may be included in agency grant proposals and annual reports. An unfavorable or neutral finding may be used by the agency to make necessary changes to improve client outcomes.

   If the team has been able to secure approval from more than one agency, it must then decide which agency to go with. This decision should be based on feasibility, that is, which agency has the best, most accessible data or from which agency would it be easier to obtain data?

   If you have neither an internship or job, then this will have to go to someone else on your team. Another possibility is to ask someone at your prior internship if the agency would like to share data or have you do an evaluation.  If no other options are available to you, you may use the virtual agency with data at: http://humanservicesnj.weebly.com/(Links to an external site.)Links to an external site.

2. Should we be evaluating one program within the agency?

   As most agencies offer more than one service and often have a number of different programs, it is best to pick one program or you can pick one service offered by a program. For example, a program for battered women may offer job counseling, job placement, group therapy, parenting skills classes, and so forth. You could pick one of these services to evaluate.

3. Are we collecting the data, or are we using data from an evaluation that the agency has already done?

   Unbelievable as it may seem, many agencies do not collect good data and some have minimal data, if at all. For example, a mandated 12-week counseling program for batterers may only keep track of attendance. When the batterer has completed 12 weeks of counseling, he is released from the program. The problem is that no pretest was done before the counseling and no posttest was done after the counseling ended, so the program cannot say whether the batterer is changed in any way. All it can say is that he completed the program. As you may not have time to do a pretest-posttest design, you could do an outcome evaluation of such a program by doing a posttest only design. In this case you would collect your own data. Or you could do a quality assurance evaluation comparing that agency’s operations to national standards for that type of a program.

   In some agencies, there is data available that you can use. Sometimes students have to review client files to obtain the data they need. The important thing is to get data that you can enter into Winks and analyze. This means that data that has already been aggregated for a report to the state or the feds would not be acceptable. You must be able to analyze the data yourselves, not use someone else’s analysis.

4. If we are looking at a study that has already been done, should we expect that the agency will give us the data?

   If a study has already been done, it would not be acceptable for your paper unless you had access the raw data and were able to analyze it. Sometimes students can use data that has been used for a report, but are able to disaggregate the data. That is, they are able to compare clients, not by name, but by demographic characteristics such as age, sex, race/ethnicity, marital status, income, educational level, etc.

   In no instance do we need confidential information about a client, but demographic information is important for comparative purposes. Suppose we wanted to evaluate the reason that an agency has a high dropout rate, or we wanted to compare successful clients with unsuccessful ones. We would need demographic information to do this.

   In my experience, most of the time agencies are willing to share data with students as long as the agency is assured that no confidential information would be revealed. You can tell the agency representative that you speak to that no one will see the report except the Professor. Agencies are also willing to share data if they think that useful information will be gained, so if an agency gives you data, they will probably want a copy of your report in exchange.

5. Does the name of the agency being evaluated have to be revealed?

   The agency’s name does not have to be revealed, it can be kept confidential. But most of the time, agencies do not mind if the name is used as no one else will see the report except the Professor.

6. What types of evaluations are appropriate?

   The optimal, best type of evaluation is an outcome evaluation because it tells whether the agency is reaching its goals. However, there are other types of evaluations that can be very useful to an agency including the following:

   • Formative evaluation
   • Process evaluation, including quality assurance
   • Needs assessment
   • Client satisfaction
   • Cost assessment