New insights and opportunities
Web scraping is relatively inexpensive, breaks down job listings by subject area, and pulls data twice a week. Because job listings are available this frequently, they provide clear, nuanced, and timely signals about a school's staffing needs.
The importance of timeliness cannot be overstated, especially given traditional data reporting and state legislative schedules. In Washington state, lawmakers finish their work in late April or even earlier each year. For example, in spring 2023, state funding and policy decisions aimed at addressing teacher staffing challenges for the 2023-2024 school year would have been based on 2021 data (the May 2021 report on teacher shortages and the state education departmentās 2021 report card) and early 2022 data (the 2021-2022 National Teacher Shortage Survey). Job advertisement data would have provided a detailed snapshot for fall 2022 and would have been up to date for nearly a year.
The time-bound nature of the collected job advertisement data also sheds light on schoolsā and districtsā hiring processes as they relate to teacher quality. Research has shown that late hires, which occur in late summer or early fall, can have a negative impact on student learning. Observing when school districts advertise their jobs can reveal whether they hire early, late, or somewhere in between. That difference can mean new teachers have months to prepare or are struggling to get their footing once the school year begins. Such information can inform policy and targeted support for more effective management practices.
Job advertisement data also serve as an immediate gauge of policy impact. State personnel data can tell us when new hires are made, but that data is typically not available until the following school year. This lag is a constraint when policymakers face immediate problems, such as the looming expiration of federal COVID-19 relief funds. For example, another analysis looked at job advertisement data in Washington state to examine the impact of federal ESSER funding on school staffing. That analysis found that 12,000 school staff and about 5,100 teachers across the state were hired with ESSER funds. This suggests that school districts will likely see budget cuts when the funds run out.
The collected job advertising data does not capture everything education leaders and policymakers need to know. It does not tell us the percentage of teachers working outside their field of expertise or with emergency certification, nor does it reveal trends in teacher retention, diversity, or quality. However, this data serves as an important supplement to traditional data collection.
Many states are now investing millions of dollars to support new pathways into the teaching profession, and scraped data can help them target investments to the subjects and types of schools that need new teachers the most. They can also target incentives such as loan forgiveness, bonuses, housing allowances, and salary increases to school districts that are struggling to hire enough staff. Such policies have helped address shortages in the past. For example, Hawaii saw a 32% reduction in vacancies after implementing a $10,000 bonus program for new special education teachers, as one of us (Rodi Theobold) found in our research. States can also use scraped data to work with undergraduate institutions to recruit students in shortage fields, such as STEM, to earn teaching credentials. Moreover, typical counts of unfilled classroom positions tend to overemphasize the shortage of elementary teachers, who make up the largest proportion of the workforce, and underestimate the immediate need for special education and STEM teachers, who are underemployed overall.
Common sense and job posting data tell us that teacher shortages are multifaceted and complex. Based on subject area and school type, there are multiple distinct challenges to getting qualified teachers into the classroom, and different inputs and incentives are required to hire enough qualified candidates. Web-scraped job posting data highlights inequities in teacher demand and supply beyond those captured in administrative data and appears to be a useful signal of the hiring needs of school districts and schools. This low-cost, timely data collection method provides important information for policymakers and educators who want to address systemic inequities in public schools.