17 Evaluating a Rental Assistance Program in Massachusetts

Amanda Lee
Research Manager
J-PAL North America, Massachusetts Institute of Technology

This chapter describes challenges and considerations faced by an early stage and in-progress project. It was first published on August 2022 and is not included in the January 2021 print version.

17.1 Summary

Homelessness is one of the more extreme outcomes of poverty and inequality. In the United States in 2019, more than 500,000 people experience homelessness on a given night and 1.4 million people pass through emergency shelters each year (Binder 2019). Each year, significant US federal and local financial resources are devoted to combating homelessness, with direct federal expenditures totaling around $6.1 billion annually and local jurisdictions spending billions more. Many states have set up programs to provide cash assistance to help reduce homelessness and promote housing stability. If these programs are proven effective, they may be a highly cost-effective way to address homelessness. Traditional survey-based research may face important methodological problems because attrition in populations at risk of homelessness is very high. Therefore, administrative data (e.g., employment and tax records) is the best way to get a great answer to these questions.

For households facing eviction, foreclosure, or other housing emergencies, the Massachusetts Residential Assistance for Families in Transition (RAFT) program provides up to $10,000 per household to preserve current housing or move to new housing (Mass.gov n.d.a).310 Researchers at the Harvard Kennedy School (HKS) partnered with the Massachusetts Department of Housing and Community Development (DHCD) to evaluate the effect of RAFT on housing outcomes (housing status, eviction, and others) as well as a broader range of outcomes including employment, income, public assistance receipt, and potential downstream effects such as those on the education of children in families.

Administrative data is key both to identifying the population of interest (those potentially facing housing instability) and measuring the variety of outcomes listed above. Critically, each of these steps requires administrative data that the DHCD (which runs the RAFT program) does not own. In Massachusetts, each agency holds its own data. Therefore, even to share data with other agencies, a data use agreement (DUA) must be established for each instance of sharing. This structure is not unique to Massachusetts, and similar difficulties in sharing data can be found in many other states and countries.

J-PAL North America works with state and local governments and researchers to develop and support randomized evaluations. Often, a key part of this support is setting up DUAs between multiple stakeholders (researchers and agencies as well as between multiple agencies). One of the main goals of the Innovations in Data and Experiments for Action Initiative (IDEA) is to support the ongoing development of partnerships with governments and other data providers that successfully use administrative data for decision-making and evaluation. HKS professors Will Dobbie and Desmond Ang and their partner Adam Schaffer at the DHCD received support from IDEA to develop a pilot evaluation and solidify a strategy to establish the required data-sharing agreements.

Beyond solving these challenges for a single study, the goal of the North America IDEA pilot projects is to work toward building longer-term partnerships with data providers and implementing partners by developing frameworks and public goods that will make not only this project possible but also future research with the same partner more feasible. Given the desire to build long-term research partnerships, J-PAL North America identified Massachusetts as a high-value partner given the Commonwealth’s significant programmatic and data responsibilities, interest from multiple J-PAL-affiliated professors in doing research with the state, including two other projects that J-PAL North America has supported (Loeb, Meyer, and Madison n.d.; EMPath n.d.), and real but solvable challenges to using data for research. The author, a research manager at J-PAL North America, provided support during this process, primarily around strategy to ensure access to necessary administrative data.

This case study highlights generalizable lessons for data sharing across government agencies, as well as with researchers, to conduct a randomized evaluation. It will explore the benefits of partnering with a state agency, lessons learned about how agencies in Massachusetts have previously succeeded in sharing data with each other, the importance of the legal context, and design challenges as they relate to data access and sharing.

17.2 Introduction

17.2.1 Motivation and Background

The conception of this project was largely driven from within the DHCD. The DHCD was, and has remained, one of the main driving forces behind this evaluation. It was already working with J-PAL-affiliated researchers through a technical assistance relationship. To further develop this partnership into a randomized evaluation, the DHCD applied to J-PAL North America’s Housing Stability Evaluation Incubator (J-PAL North America n.d.), a practitioner-facing program designed to develop housing stability-related research ideas into feasible evaluations. As part of the incubator, the DCHD, Ang, and Dobbie worked to further develop an evaluation. Because administrative data access was a primary challenge for conducting the evaluation, J-PAL selected the project to receive IDEA support, funded by the Alfred P. Sloan Foundation. Even with this outside support, continued internal agency leadership would prove important, both in defining the type of project and in opening doors for initial discussions with other state agencies.

When the research team (the DHCD, researchers, and J-PAL staff) began designing the evaluation, everybody who applied to RAFT and was eligible received funds. That is, funding for the program was not a constraint. The research team recognized that it would be unethical to design a randomized evaluation in which RAFT-eligible households would be randomized into a group that did not receive RAFT. Therefore, to proceed with a randomized design that allows for estimating causal effects of the program, the team decided to use a randomized encouragement design,311 where the treatment group was provided additional encouragement to sign up for the program and the control group could sign up for the program based on existing information. Both groups could sign up for services.

In an encouragement design, the research team first identified a group of people likely eligible for RAFT based on their income. In Massachusetts, households who receive benefits through the Department of Transitional Assistance (DTA) have incomes that would likely make them income-eligible for RAFT. From this pool of eligible participants from the DTA, the research team randomized them into two groups: the treatment group would receive information about RAFT and support in completing the application, and the control would not receive any additional information about RAFT. Households in both groups could apply for RAFT, but the encouragement, if effective, was supposed to produce higher levels of enrollment into RAFT in the treatment group.

To simplify the initial data-sharing process and to develop the encouragement mechanism, the research team decided to run a pilot study that would measure whether and how much the encouragement increased enrollment in RAFT (the “first stage”) before designing an evaluation of the effect of RAFT on downstream outcomes, such as income, employment, and housing. This was done because in an encouragement design, it is important for there to be a large enough difference in RAFT enrollment rates to allow a comparison between groups. In other words, the pilot would help determine whether the full study had sufficient statistical power.312 As of October 2021, the pilot had not started. If the information about the program and application assistance do not help more people enrolled in RAFT, then the research team will want to consider changing the encouragement before attempting a comprehensive study.

17.2.2 Data Use Examples

Administrative data needs differed between the pilot and the full study. These differences drove the prioritization of conversations and the structure of agreements. Running the pilot meant the study team only needed immediate access to some data and could defer some of the more complex data discussions. For the pilot, the DTA held data critical for targeting the intervention: a list of households receiving DTA services who were likely eligible for RAFT. Pilot outcome data about whether participants enrolled in RAFT is held internally at the DHCD.

For the full study looking at the impacts of RAFT, the DTA, along with multiple other agencies, hold outcome data of interest. For example, the research team was interested in receiving wage and employment information from the Department of Revenue (DOR) and Department of Unemployment Assistance, student test scores from either Boston Public Schools or the Department of Elementary and Secondary Education, and health outcomes from the Department of Public Health, in addition to public assistance receipt from the DTA.

Splitting the work into these two phases meant that the pilot was a much easier lift in terms of administrative data-sharing requests. Still, although the additional outcome data sources are not needed until later, an understanding of what will be feasible for the later study is helpful now since the pilot is designed to inform that future work and would be less impactful on its own. Splitting the work led to a strategy of developing an initial agreement for data sharing with the DTA while having concurrent conversations with agencies about future data needs, as discussed in more detail below.

17.4 Protection of Sensitive and Personal Data

17.4.1 Safe Projects: Evaluating Data Analysis Projects for Appropriateness

The DHCD does not have its own institutional review board (IRB) or research review process. The decision around whether to share data was focused around (1) whether it could be used for research, (2) some discussion of data safety, and (3) whether the work and resources required fit within an agency’s current priorities and resources.

Harvard University’s IRB determined the pilot to be exempt from IRB review because it was for program improvement of RAFT, one of the DHCD’s own programs. For the pilot, identified data will not need to leave DHCD servers, researchers will not need to see identified data, and the person doing the encouragement will be a DHCD employee. In addition, for the pilot, the research team is not requesting outcome data from any other agencies and expects the full study to require IRB approval through Harvard University.

While the IRB determination may not have influenced decisions to share or not share data in this instance, this type of reasoning could be useful in allowing agencies to share data. Sometimes, sharing data across agencies for “program improvement” can make it easier to share data rather than for “research.” In fact, demonstrating some benefit of the research to program administration is often a requirement for data access even among government agencies that have established research data access procedures.

17.4.2 Safe People: Evaluating Researchers Who Seek Data Access

As discussed above, state agencies did not appear to have preferences or requirements as to whether researchers needed to be “special employees,” which would essentially allow them to be government employees for the purpose of data access. Moreover, there appears to be no standardized vetting process for researchers wanting to partner on research projects with the state or access data. In our particular case, the researchers had an existing relationship with the DHCD through a longer-term technical assistance partnership, external to J-PAL, that predates this research engagement.

In all discussions we have made it clear the project would limit access to researchers and agency staff who are necessary to perform data transfer and analysis (i.e., “safe people”). This group of people may vary depending on where the matching and linking of data occurs (e.g., if agencies allow researchers to link data, they will view de-identified data, but if the agencies perform the matching, researchers will only view de-identified data).

17.4.3 Safe Settings: Accessing Data

Data access and sharing procedures have not been finalized. The researchers plan to use secure data storage and access procedures in accordance with their institution and any additional requirements from the DHCD and other data partners whose data may be used as part of this study. In early discussions with agencies, it seems likely that Harvard University’s requirements313 for data security and access are as stringent if not more stringent than those required by various agencies. In discussions for the full study, we anticipate that the university’s security measures will help reassure agencies that researchers would treat data appropriately. Data access procedures will be outlined in the IRB protocol and in DUAs, and access to data will be limited to safe people, as defined above.

17.4.4 Safe Data: Verifying and Mitigating Disclosure Risk

Researchers will minimize disclosure risk by using de-identified data when possible, and perform linking or matching in the safest way possible, in this case in accordance with Harvard’s Research Data Security Policy (Harvard University 2020b). For example, if a data agency has internal capacity, it can perform the link itself and share de-identified data with the researchers for analysis. This type of setup has lower disclosure risk since it prevents any identified information from leaving the agency. If agencies do not have the capacity to perform the match, then researchers can set up secure data transfer methods. Harvard classifies types of data based on their data security level (Harvard University n.d.); with each higher level comes more identified information shared and more protection protocols in place. There are many different data linking and sharing scenarios depending on researcher and agency capabilities; researchers will explore and choose the safest possible feasible approach to answer the research question.

17.4.5 Safe Outputs: Verifying and Mitigating Disclosure Risk in Statistical Analysis, Results, and Tabulations

The exact review processes for disclosure risk will be determined when the full study begins. Researchers will follow rules agreed upon in DUAs and follow standard practices such as suppressing results from small cells and not publishing identifiable information.

17.4.6 Data Life Cycle and Replicability

As of May 2022, researchers have not yet gained access to data, and therefore the preservation and reproducibility of both researcher-accessible and researcher-generated files has not been established. Given that the primary researcher-accessible files we are interested in are based on benefit eligibility and receipt across various programs, which is something agencies calculate on an ongoing basis, the reproducibility of these types of files seems high. This will be a topic of future discussion. Researcher-generated files will follow best practices (e.g., writing code to automate and ensure the reproducibility of data cleaning and analysis).

17.5 Sustainability and Continued Success

17.5.1 Outreach

The project to date demonstrates successful outreach on the part of the DHCD, HKS researchers, and J-PAL to generate this research partnership. The partnership between the DHCD and Ang and Dobbie began as a technical assistance engagement through the Government Performance Lab at the HKS. The team continued to look for additional support in applying to J-PAL North America’s Housing Stability Evaluation Incubator, which ultimately led to support through IDEA. Both the DHCD and the researchers have shown commitment to bringing in additional collaborators and working together to achieve shared research objectives. In addition, a better understanding of how data is created, stored, and shared at Massachusetts agencies allowed the research manager to reach out to the DTA and DOR regarding a separate research project. Successful research partnerships require commitment from multiple parties and offer the hope that lessons learned will be beneficial in repeat interactions and longer-term relationships.

17.5.2 Revenue

The data access mechanism, the number and type of data sets used, and the length of time for access have not been established. Therefore, we are unable to comment on the financial stability at this time.

17.5.3 Metrics of Success

Data access is still in progress. A DUA for the pilot — with reassurances for the full study — is in the process of being developed. It has not been signed as of October 2021. Pending the DUA’s execution, gaining access to data so far appears at least partially successful. Signed agreements, launching the pilot, and agreements for data use for a full study are the next measures of success for this project.

About the Author

Amanda Lee is a research manager at J-PAL North America. She provides research support across multiple randomized evaluations and manages the Catalog of Administrative Data Sets.

Disclaimer

This case study reflects the experiences of the author on this particular project and does not necessarily represent the views of the DHCD or any other agency of the Commonwealth of Massachusetts.

References

Binder, Jacob. 2019. “Reducing and Preventing Homelessness: Lessons from Randomized Evaluations.” J-PAL Evidence Review. https://www.povertyactionlab.org/sites/default/files/publication/rph_homelessness-evidence-review.pdf.

Doyle, Mary-Alice, and Laura Feeney. 2021. “Quick Guide to Power Calculations.” The Abdul Latif Jameel Poverty Action Lab (J-PAL). https://www.povertyactionlab.org/resource/quick-guide-power-calculations.

EMPath. n.d. “AMP up Boston.” Accessed March 29, 2022. https://www.empathways.org/direct-services/amp-up-boston.

Harvard University. 2020b. “Research Data Management.” https://research.harvard.edu/2020/06/26/research-data-management/.

Harvard University. n.d. “Data Security Levels.” Accessed July 19, 2022. https://policy.security.harvard.edu/view-data-security-level.

Heard, Kenya, Elisabeth O’Toole, Rohit Naimpally, and Lindsey Bressler. 2017. “Real-World Challenges to Randomization and Their Solutions.” J-PAL North America.

J-PAL North America. n.d. “Housing Stability Evaluation Incubator.” Accessed March 29, 2022. https://www.povertyactionlab.org/HousingStabilityEvaluationIncubator.

Loeb, Susanna, Katharine Meyer, and Samuel Madison. n.d. “The Impact of Text Message Nudges on Churn in the Supplemental Nutrition Assistance Program in the United States.” J-PAL Evaluations. Accessed March 29, 2022. https://www.povertyactionlab.org/evaluation/impact-text-message-nudges-churn-supplemental-nutrition-assistance-program-united-states.

Mass.gov. n.d.a. “Emergency Housing Payment Assistance During COVID-19.” Accessed March 29, 2022. https://www.mass.gov/info-details/emergency-housing-payment-assistance-during-covid-19.

Mass.gov. n.d.b. “Learn to Earn Initiative.” Accessed March 29, 2022. https://www.mass.gov/service-details/learn-to-earn-initiative.


  1. During the pandemic, Massachusetts also operated the Emergency Rental Assistance Program, which provided up to 18 months of support for rent arrears and future rent stipends.↩︎

  2. See Heard et al. (2017) for a definition and discussion of use cases for an encouragement design.↩︎

  3. Because participants in both groups can enroll in RAFT, conducting a randomized evaluation requires a sufficiently large, randomly induced difference in participation as a result of the encouragement in order to be able to detect effects. This “first-stage” relationship between encouragement and enrollment is critical to having a sufficiently powered study. Differences in outcomes are estimated by comparing the entire treatment group to the entire control group regardless of enrollment. RAFT, however, can only help those who enroll. Estimates of its impacts are diluted by individuals who are treated by the encouragement but do not enroll and by a lack of difference in enrollment between the treatment and control groups. Thus, conducting the study requires that the program be undersubscribed at baseline (which is true) and the encouragement sufficiently increases enrollment (which we will learn from the pilot). How much the encouragement increases enrollment will be used to calculate the sample size needed for the full study. Power considerations for encouragement designs are discussed in Heard et al. (2017), and Doyle and Feeney (2021) provide a guide to power calculations.↩︎

  4. Harvard University (2020a) https://policy.security.harvard.edu/view-data-security-level↩︎