The following information will assist you when using these questions to guide decision making:
- The State Performance Plan
- The Annual Performance Reports
- Letters of Determination with accompanying Tables
- The state sampling plan, if sampling for Indicator 14 data
- If using a contractor, the contract agreement and any reports
- District level Indicator 14 post-school outcome data (including nonresponse rate) disaggregated by gender, ethnicity, age, disability type, and method of exit. This means that you will have data that will tell you the rate of employed, enrolled in post-secondary schools, or both for each group (i.e., for gender: males and females).
Who manages data collection efforts: a contractor, the state, or district personnel?
- Is your state contracting with an organization or university to collect, analyze, and report the data?
- Is your state education office directing the collection, analysis, and reporting of the data?
- If contracting, who oversees the contract?
- What state resources are currently allocated to fulfill the Indicator 14 requirements?
- Who manages the internal system to collect, analyze, and report the data?
- Are your transition, data, and monitoring personnel working together to manage the system?
What leavers are in the target group of leavers: a representative sample or census?
Should you conduct a census of all school leavers in your state?
- Can you identify all school leavers (i.e., graduates, dropouts, age outs, and those were expected to return, but did not) in your state for the academic year?
- Do you have the resources to identify and gather information on all school leavers’ demographics, in-school transition experiences, contact information, and post-school transition experiences?
IF “NO” TO EITHER OR BOTH OF THE ABOVE QUESTIONS, IT IS PROBABALE THAT YOUR STATE WILL NEED TO ESTABLISH A REPRESENTATIVE SAMPLE ON WHICH TO GATHER DATA FOR INDICATOR #14.
Sampling Considerations
- How do you address the following requirements
- School districts of 50,000 students (or more) must be included in the sampling yearly?
- Every district in your state must be included in the sample at least once over the course of your SPP?
- Have you established the sample by sampling students or districts?
- Have you submitted your sampling plan to OSEP for approval? (OSEP needs to approve all sampling plans.)
To increase the validity and reliability of your data, how are you establishing a representative sample of school leaver’s with disabilities on the following student characteristics?
- Is the sample representative (i.e., +/- 3%) of all school leavers in terms of how the students with disabilities exited high school – i.e., graduated with a regular diploma or an alternative/modified diploma, dropping out, and aging out?
- Is the sample representative of all disability categories?
- Is the sample representative of ethnic/racial groups?
- Is the sample representative of ages?
- Is the sample representative of gender?
- How are you testing the similarities or differences of the respondents to the target group of leavers on each of these categories?
Post-School Outcomes Data Collection
HOW are the data being collected?
- Is your state using extant data (e.g., Department of Labor, Department of Corrections, higher education, etc.) to gather post-school outcome data? If so:
- Are relevant databases containing status information on post-school outcome available within your state?
- Are appropriate cross agency sharing of information agreements in place?
- Are unique student identifiers used that connect in-school and post-school databases?
- Are the variables within these databases defined so they provide the needed information for reporting post-school outcome data for your SPP/APR and for state and local use?
- Do you or your contractor have the technical expertise to merge, match individuals across merged databases, and analyze the data?
- Is your state using survey methodology to gather post-school outcome data? If so:
- Have you defined a specific type or combination of survey data collection methods (e.g., phone, web-based, mail, etc.)?
- Have you developed a data collection protocol and piloted it with former students and/or parents from a variety of disabilities to increase the validity of your instrument?
- Have you defined respondents (e.g., former student, parent, student’s designee)?
- Have you defined strategies to ensure high response rate, specifically for the hard-to-find populations (e.g., pre-contact notice, multiple appropriate respondents, appropriate accommodations for survey type, etc.)?
- Have you defined a procedure to calculate response rates?
- Is the data collection survey available in alternate forms (Braille, different languages, etc.)?
- If interviewing, how will you collect data from former students who use sign language or an alternate form of communication?
To increase the reliability in the data collection process, WHO collects the data?
- Is state, district, or school staff responsible for data collection? If so,
- Who are the data collectors: (e.g., former teachers, support staff)?
- Has adequate technical assistance and training been provided to those collecting the data?
- Has a procedure been set in place to ensure trained individuals are following the data collection process with fidelity?
- Is a contractor responsible for data collection? If so,
- Has the scope of work, with timelines and definitions, been clearly defined and discussed with the contractor?
- Are appropriate sharing of information and confidentiality procedures in place?
- How will data transfer to and from the contracted agency be executed?
- Has a procedure been set in place to ensure that valid and reliable data are collected by the contractors? (e.g., Were data collectors trained? Were data collected with fidelity?)
To increase the reliability of your data, WHO are data collected on?
- What procedures are in place to identify and collect exiting information on individuals who age-out or drop out of school?
- What procedures are in place to collect data, one year after exit, from school completers (i.e., graduates) as well as from individuals who have aged-out or dropped out of school?
To increase the validity of the data, WHAT data are collected?
- What exiting information is collected prior to a student’s exit (e.g., multiple contact information, demographic, completion status)?
- How is competitive employment defined?
- How is post-secondary school defined?
- Is your data collection protocol designed to collect the information required for federal reporting?
- What variables have you identified that will be useful for program improvement and decision-making at both the state and local level?
- What other variables (i.e., independent living and social adjustment) are needed for program improvement?
WHEN are data collected?
- In-school data collection:
- When is in-school data collected for completers & non-completers?
- Are you collecting the minimum demographic data (e.g., gender, disability type, race/ethnicity, method of exit, age)?
- What additional information is needed to link Indicators 1, 2, & 13 (e.g., postsecondary goals, IEPs goals, classes taken, etc.)?
- Do you collect multiple forms of contact information (e.g., home address, cell, e-mail for student, family, and best friends)? Is this updated annually?
- Do you capture follow-up contact information for dropouts before they become dropouts (i.e., if there is a formal dropout procedure, does that include contact information for other family members, best-friends, etc.)?
- Post-School data collection:
- Do you collect post-school outcomes between April through September for all school leavers?
- If not, have you received approval from OSEP?
Post-School Outcomes Data Analysis and Federal, State and Local Reporting
Have you tested for representativeness of your respondent group to your target leaver group (either your representative sample or total population)?
- How will you test how similar or dissimilar respondents (e.g., +/- 3 percentage points) are to the target leaver group relative to age, gender, disability, ethnicity, and exit type (e.g., dropout)?
- Have you used the NPSO Response Calculator to test representativeness?
Public Reporting
- In your SPP/APR do you provide a description of your data collection procedures, including:
- How data are collected on census or representative sample of school leaver population;
- If sampling, describing the process and how the target group is representative of the population of school leavers;
- Who collects the data, school staff, organization hired by state/LEA;
- What time of the year the data are collected, (month(s) of year) (i.e., April through September);
- How data are collected, (e.g., phone and/or mail survey, face to face interviews);
- Who is interviewed (e.g., former student or designee);
- What methods, beyond the state website, are used to report to the public about the performance of each local agency in our sample (e.g., newspaper, booklet, etc.);
- Baseline percent of those competitively employed, enrolled in postsecondary education/training, or both, discussion of baseline, targets, slippage/progress, and improvement activities?
- Have you put safeguards in place not to report in personally identifying information (i.e., are numbers of less than 10 in a reporting cell not publicly reported?)
- Do you coordinate data collection, analysis, reporting, and improvement activities across indicators 1, 2, 13, and 14?
Reporting Considerations to facilitate the usability & accessibility of the data
- Who are the audiences that need information about post-school outcomes for students with disabilities?
- How will you communicate the results to legislators, labor market audiences, district and local educators, teachers, parents, and youth with disabilities- former and current students?
- What information do these audiences need and want (i.e., for programmatic decision-making)?
- What do you expect them to do with the information (e.g., make decisions regarding program continuation or expansion, evaluate affects of policy and procedures, or take no action--just be aware of the percent of youth working, going to school, or both?)
- What method of presentation best meets the needs and purposes of the targeted audiences:
- graphs and bar charts
- trend lines
- numeric tables
- other graphics
- descriptive narrative
- For graphic presentation of the data, do you include:
- Bar graphs with contrasting patterns and colors (light/dark) that can be read easily after being copied multiple times and still interpreted in black and white?
- A percentage or number at the top of each bar to help readers grasp the parentages/numbers being represented?
- A legend large enough to read after being reduced or duplicated?
- A concise narrative statement describing and summarizing the data to help readers understand the report (e.g., what was measured, who was interviewed, when, key findings)?
- Definitions for the terms used (e.g., exiters, school leaver, competitive employment and postsecondary education/training enrollment?
- The same scale and metric on all graphs?
- For narrative presentation of the data, do you include:
- The purpose, participants, and procedures described clearly and concisely?
- A summary of the key findings?
- Italics and bolding to draw attention to important results, without over using them to detract from the message?
- Definitions of frequently used abbreviations and acronyms?
- Other considerations for data use and reporting:
- Do you provide short (“just the essential facts”) reports to all local agencies with further data sets as requested?
- What training will your personnel need in order to understand and use the results at the state and local level?
- Do you need to provide technical assistance to local education agencies and community leaders to help them understand and use the data?
- Do you examine post-school outcome data & findings in relation to graduation, dropout, student performance on assessments, IEPs, and transition services?
- Do you report post-school outcomes data in relation to Indicators 1, 2, 13, and 14?
- For each method of reporting, what adjustments are needed to accommodate a wide range of individuals’ abilities and disabilities?
- Is personally identifying information protected?
Using Post-School Outcome Data for Program Improvement
Questions to facilitate using the data for programmatic decision-making
- In examining your data, the following questions can help define targeted strategies to improve post-school outcomes:
- Were specific sub-groups or respondents not representative of their respective target group specifically on age, gender, race, and disability type? (i.e., were sufficient dropouts included in your data? Are the race/ethnic groups in your state similar or dissimilar to your target group?) If so,
- Are cautions stated when reporting such data on various sub-groups that were not representative?
- Are procedures in place to increase the reliability of the data collection next year, to eliminate this problem in future data collections?
- What sub-populations did not meet or exceeded your state target?
- Did males and females perform the same for employment and/or post-secondary education/training?
- Did each disability type perform the same for employment and/or post-secondary education/training?
- Did each age group perform the same for employment and/or post-secondary education/training?
- Did each ethnicity/racial group perform the same for employment and/or post-secondary education/training?
- Did each leaver type (i.e., high school diploma recipients, modified diploma recipients, drop-outs, age-outs) perform the same for employment and/or post secondary education/training?
- If the answer was no to any of the above questions, then ask:
- Are there policies in place that may be contributing to the varying outcomes defined? (e.g., policies surrounding diploma types, number of academic courses required for graduation, or that prohibit or discourage off-campus work experiences)
- What services and/or evidenced-based practices are in place for those sub-groups who may be performing above the state benchmark? How can these be duplicated in the areas where sub-groups are not meeting the state target?
- What evidence-based practices are needed to support the sub-groups who are performing at a deficit to the state target?
- What professional development/technical assistance is needed to implement evidence-based practices?
Original Word Document
