Research Questions in Federal Waiver Evaluation Design

CMS is funding a national cross-state evaluation that will consider several issues related to Section 1115 Medicaid expansion waivers, including the impact of Marketplace premium assistance, premiums, and healthy behavior incentive programs.1 Key research questions identified in the federal waiver evaluation design plan2 related to these issues include:

  • Marketplace premium assistance: How do states that require beneficiaries to enroll in Marketplace Qualified Health Plans (QHPs) using Medicaid as premium assistance compare to states implementing traditional expansions using their Medicaid delivery systems, in terms of enrollment rates, continuity of coverage, access to care, health outcomes, and health care and administrative costs?
  • Premiums: What effect do premiums imposed on beneficiary groups who are not otherwise subject to premiums under federal law, premiums that exceed the limits in federal rules, and lock-out periods for non-payment of premiums have on enrollment and continuity of coverage?
  • Healthy behavior incentive programs: Do beneficiaries understand healthy behavior incentive programs? Which educational strategies are most effective? Which incentives encourage beneficiaries to actively participate in their care without impairing access to care, and which yield the greatest relative gains in preventive care and management and care of chronic conditions and the greatest reductions in emergency room use? What are the administrative costs to states and managed care plans of implementing healthy behavior incentive programs?

Appendix 1 provides a full list of federal research questions.

Unlike the individual state waiver evaluations (described below), the federal Section 1115 waiver evaluation can assess the experience across states in implementing common waiver provisions. The federal evaluation will examine Marketplace premium assistance programs in Arkansas, Iowa and New Hampshire, and premiums and beneficiary engagement (such as healthy behavior incentives) programs in Arkansas, Indiana, Iowa, Michigan, Montana and Wisconsin (these are the issues most relevant to the expansion waivers).3 The federal evaluation also will consider questions related to other types of Section 1115 Medicaid waivers, including Delivery System Reform Incentive Payment (DSRIP) programs and managed long-term services and support programs.4

Research Questions in State Waiver Evaluation Designs

The Medicaid expansion demonstrations set out fairly broad goals under which the states have identified a number of hypotheses for evaluation. While each state’s evaluation is unique, the themes of evaluation questions identified by the states relevant to the Medicaid expansion waivers can be grouped into five key areas: coverage, access to care and utilization, premiums, healthy behavior incentives, and program costs. The rest of this section summarizes the states’ waiver evaluation questions in these five areas. A full state-specific summary of evaluation goals and hypotheses that will be tested is included in Appendix 2.

Coverage

While no waiver is required to examine the implications of the Medicaid expansion on changes in coverage, Michigan’s and Indiana’s evaluations will measure their waivers’ impact on change in the number of uninsured. Both states expect to experience reductions in the uninsured as a result of their Medicaid expansions. Michigan expects that both the reduction in uninsured and the increase in Medicaid enrollment will be significant relative to expected state trends absent the Medicaid expansion, similar to the experience of other expansion states, and significantly larger than the changes in non-expansion states. Reductions in the uninsured and increases in Medicaid coverage are expected in all states that implement the Medicaid expansion; so, this issue is not specific to Medicaid expansion waivers; however, other expansions are not required to conduct evaluations so other states may not be explicitly tracking these changes.

Several states plan to examine their demonstration’s impact on continuity of coverage and gaps in care. Arkansas and New Hampshire are testing the impact of mandatory Marketplace premium assistance on continuity of coverage. These states anticipate that as a result of Marketplace enrollment, Medicaid expansion adults will experience fewer gaps in coverage and more continuous access to the same health plans and provider networks. Although no longer using a Marketplace premium assistance model as originally intended,5 Iowa’s evaluation will examine whether beneficiaries maintain access to a regular source of care when Medicaid eligibility changes due to income fluctuations. Indiana will test how, in the absence of retroactive eligibility, its waiver provisions to utilize additional presumptive eligibility and fast-track prepayment of premiums (required for enrollment to become effective) affect gaps in coverage. It is expected that Montana’s waiver evaluation will examine continuity of coverage, as it relates to the state’s adoption of 12-month continuous eligibility.

Arkansas and New Hampshire expect that their premium assistance waivers will impact the premiums and coverage options in their Marketplaces. Arkansas anticipates that providing Marketplace coverage for Medicaid expansion adults will reduce premium costs in the Marketplace. New Hampshire projects that using Marketplace premium assistance could lead to a greater variety of Marketplace health plans, as insurers will have incentives to offer both Medicaid managed care and Marketplace plans.

Indiana plans to test how its demonstration promotes access to employer-sponsored insurance (ESI) and reduces fragmented coverage within families. Indiana’s waiver includes a voluntary defined contribution ESI premium assistance program, which the state expects will increase the proportion of Medicaid expansion adults who are covered by ESI. Indiana also expects that its ESI premium assistance option for family coverage will increase the number of families in which all members have access to the same provider network instead of having parents enrolled in ESI and children in Medicaid or CHIP.

While not directly related to coverage, Indiana’s evaluation includes an assessment of its state-funded work search program, which is not considered part of the Section 1115 demonstration by CMS.

Indiana’s Work Search and Job Training Program Evaluation

Indiana’s evaluation of its Medicaid expansion waiver includes an assessment of its state-run voluntary work search and job training program. CMS did not approve Indiana’s request to require a work referral as a condition of eligibility as part of its Medicaid expansion waiver and noted that Indiana’s work search and job training program is separate from the Section 1115 demonstration. Indiana expects that Medicaid applicants who are referred to work search and job training resources available through the state’s Department of Workforce Development will have increased employment rates over the course of the demonstration.

Access to Care and Utilization

Several state evaluations will examine changes in beneficiary access to care. Arkansas, Iowa, and New Hampshire will assess changes in access to primary, preventive, behavioral health, and specialty care and Early and Periodic Screening, Diagnostic and Treatment (EPSDT) services for 19- and 20-year-olds. Iowa also plans to measure access to dental care, including preventive dental services. Arkansas will compare access to care in its Marketplace premium assistance model to access in its Medicaid fee-for-service delivery system, while New Hampshire will compare access to care for expansion adults to access for the state’s general population. Arkansas, Iowa, and New Hampshire also mention plans to assess beneficiary satisfaction or experience with care as part of their evaluations. In Arkansas and Iowa, the evaluation plans will examine the adequacy of provider networks as part of their evaluations. Iowa expects that its Medicaid expansion enrollees will have the same access to medical providers and improved access to dental providers, relative to traditional (non-expansion) Medicaid populations in the state.

States expect their demonstrations to reduce potentially preventable emergency room use and hospital admissions through a variety of measures. For example, Arkansas, Iowa, Michigan, and New Hampshire plan to assess whether improved access to primary care services for expansion adults results in lower rates of non-urgent use of the emergency room and fewer potentially avoidable hospital admissions. Michigan also expects that beneficiaries who participate in its healthy behavior incentive program (described below) will have fewer emergency room visits and hospital admissions. Indiana hypothesizes that non-urgent emergency room visits will be reduced through its use of graduated copayments, prior authorization, and expanded access to urgent care. Iowa will assess whether copayments for non-emergency use of the emergency room (at state plan amounts) decrease inappropriate emergency room use without imposing barriers to access to care.

States that have waived non-emergency medical transportation (NEMT) and those that are delivering these services as a wrap-around benefit plan to measure access to transportation services. States that have waivers of NEMT (Iowa and Indiana) will measure barriers to care resulting from lack of transportation, but hypothesize that the waivers will not create access barriers. For example, Indiana will measure the effect of its NEMT waiver on missed appointments, by income level, in various geographic areas of the state, and on access to preventive care and overall health outcomes.  The state plans to assess the impact of its NEMT waiver on both providers and beneficiaries.  Arkansas and New Hampshire do not have waivers of NEMT services but plan to measure access to NEMT, which is provided as a wrap-around service in their Marketplace premium assistance programs.

Premiums

States implementing programs to collect premiums or monthly contributions and/or cost-sharing will test whether these payments are affordable and whether they create barriers to health care access. Indiana and Iowa hypothesize that monthly premiums will not present barriers to access. Indiana will examine the number of individuals who are locked-out of coverage for non-payment of premiums, and the number of beneficiaries who have employers and/or not-for-profit organizations paying all or part of their required premiums. Michigan expects that its model, in which monthly premiums and cost-sharing based on the prior six months of service use are paid into health savings accounts, will not be associated with expansion adults dropping their coverage.

Demonstrations in Indiana and Michigan will test whether policies related to out-of-pocket contributions lead to greater beneficiary engagement in their health care and more efficient use of services. Indiana expects that beneficiaries who make monthly payments into their health savings accounts will exhibit more cost-conscious health care consumption than other Medicaid beneficiaries without harm to their health. Indiana also hypothesizes that its provisions for rollover of health savings account funds and coverage of preventive care without copayments will encourage beneficiaries to make monthly account payments and actively manage their account funds. Michigan expects that cost-sharing will lead to more efficient health care utilization when comparing beneficiaries’ total health care costs over time with costs in their first year of enrollment. Michigan hypothesizes that those above poverty, who are subject to both premiums and copayments, will use services more efficiently than those below poverty, who are subject only to copayments. As described below, efficient use of care would mean less emergency room and inpatient care and more primary and preventive care.

Healthy Behavior Incentives

Iowa and Michigan will evaluate whether their healthy behavior incentive programs lead to improved health outcomes for beneficiaries. Iowa anticipates increased utilization of annual examinations, smoking cessation services, and preventive dental services between the first and second years of its demonstration. Additionally, Iowa expects that over half of its expansion adults will earn access to enhanced dental benefits by completing specified healthy behavior activities. Michigan projects improved health over time from its healthy behavior program, as reflected in health risk assessment results, use of preventive care, adherence to medications to manage chronic disease, and self-reported health status.

Iowa and Michigan also will assess beneficiaries’ understanding of healthy behavior incentive programs. For example, Iowa will track which activities beneficiaries complete and which characteristics (such as age, race, gender, geographic area, and provider engagement with the program) are predictive of completing healthy behavior activities. In addition, Iowa and Michigan will assess beneficiaries’ understanding of the purpose of the program and how it works. Michigan plans to identify the factors that facilitate beneficiary completion of healthy behaviors and those that serve as barriers. Both Iowa and Michigan also plan to examine the impact of their healthy behavior programs on providers, such as whether providers use information from beneficiary health risk assessments and whether they have changed how they communicate with or care for beneficiaries as a result of the demonstration.

Program Costs

While longstanding federal policy requires Section 1115 demonstrations to be budget-neutral to the federal government, state waiver evaluations also will measure program costs. For example, Indiana expects its demonstration to be budget-neutral for the state as well as the federal government and will compare its waiver expenditures to expenditures in other states. Iowa anticipates that the costs for its demonstration will be comparable to the predicted cost of covering expansion adults in its traditional Medicaid program. Arkansas and New Hampshire expect that their Marketplace premium assistance models will be cost-effective compared to covering expansion adults under their traditional fee-for-service Medicaid programs.6

The Michigan and Indiana evaluations will also analyze uncompensated care costs. These states expect significant reductions in uncompensated care costs relative to pre-expansion trends. Michigan expects that hospitals with above-average baseline levels of uncompensated care costs and uninsured patients will experience greater percentage decreases in uncompensated care costs as a result of the expansion. The state also anticipates that its uncompensated care costs will decrease significantly relative to non-expansion states, and that its uncompensated care costs will not significantly differ from those other expansion states. Indiana will measure how its waiver of retroactive coverage impacts uncompensated care costs.

Key Evaluation and Research Challenges

While it is important to understand states’ experience with implementing current expansion waivers to help inform future program directions, some key challenges that will hamper the ability to conduct meaningful evaluation of the Medicaid expansion waivers.7

Data Limitations

Robust evaluations depend on access to reliable and timely data, which may not be available within the timeframes for the state and federal waiver evaluations. All of the evaluations include quantitative data analysis, but there are often lags in the availability of federal survey data that is needed for the evaluations. The federal evaluation design plan also points to potential problems with the availability of administrative data from the national Medicaid data systems. States will soon transition from the Medicaid Statistical Information System (MSIS) to a revamped version known as Transformed MSIS (T-MSIS). During the transition, MSIS data may be incomplete as states shift to reporting in T-MSIS, but T-MSIS will likely need start-up time before its data are considered to be reliable. Some states plan to rely on state surveys or surveys of beneficiaries; however, state surveys can be expensive to administer and data from the surveys may not be comparable to data in other states to enable cross state comparisons.

Selection of Outcome Measures and Interpreting Data

Selecting outcome measures and interpreting evaluation findings can be extremely complicated. The selection of outcome measures to study in evaluations of Medicaid expansion waivers and the interpretation of data findings are both important challenges. Some outcome measures, such as change in the coverage status of low-income adults and change in out-of-pocket costs for health care, are more direct measures of the impact of Medicaid expansion. However, other expected downstream impacts of increased coverage, such as increases in access to care and improvements in health status, may be mediated by other individual-level or system-level factors, take a longer time to materialize or not materialize, or simply be difficult to interpret. Conflicting interpretations of early findings about access to care and health outcomes from the seminal (pre-ACA) Oregon Health Insurance experiment highlight issues associated with evaluating the impacts of Medicaid expansions, related to study design, methodological choices, timing, and other matters.8,9,10

Generalizability

Small sample sizes and unique state circumstances make it difficult to generalize findings from state evaluations to broader populations. One of the goals of Section 1115 waivers is for states to identify new approaches that may be effective elsewhere. However, evaluation results may not be generalizable because some state evaluations call for comparisons of Medicaid expansion adults to other populations in the state. Each state has a unique set of factors from the Medicaid delivery system to broader health care market factors. So, even evaluations that show robust results for a specific state may not be generalizable to other states due to these specific state circumstances.

Administration

The waivers’ administrative complexity makes it difficult to interpret evaluation findings.  For example, in states that assess a premium, there are often options to mitigate the effect of the premium for beneficiaries.  In Indiana a third party may contribute to the premiums payments which may may make it hard to determine the effect of premiums on low-income individuals.  Many aspects of the expansion waivers are extremely complicated for states to administer, and in some cases, states have delayed implementation of certain provisions. This administrative complexity could cause provisions to be implemented in ways that differ from what was originally intended. For example, Indiana’s implementation of the lock-out provision for non-payment of premiums was delayed; Arkansas received approval to implement monthly contributions for individuals down to 50% FPL but, due to administrative costs has only implemented the policy for those above poverty, to date. Sometimes, these changes are course corrections where a state is able to assess available data and information to assist in the implementation process.

Most states are not specifically considering administrative costs as part of their waiver evaluations, despite the programs’ complexity. Only New Hampshire specifically plans to look at the administrative costs of its waiver.  It anticipates that using Marketplace premium assistance may result in lower administrative costs.  Other states are not planning to assess the impact of their waivers on state administrative costs even though their waivers often introduce complicated program elements, such as premiums, cost-sharing, and healthy behaviors, that require ongoing tracking by the state.

Dynamic Environment

It is challenging to isolate the effects of policy changes related to the expansion waivers from the effects of policy and other changes unrelated to the waivers. State Medicaid expansion demonstrations do not operate in a vacuum. Along with the expansion, broader coverage expansions, enrollment and application streamlining, and insurance market reforms are also being implemented as part of the ACA. In addition, states are simultaneously implementing a variety of other reforms, including multi-payer delivery system reforms through State Innovation Model grants, Medicaid managed care initiatives, and Accountable Care Organizations – any or all of which could affect access to care, health outcomes, and costs. In such a dynamic environment, isolating the impacts of an expansion waiver can be difficult. With multiple large-scale policy changes underway all at once, it will be hard, in both the state-level and federal cross-cutting evaluations, to attribute observed changes to particular waiver-related interventions or reforms.

States have also amended original waivers adding complexity to the evaluation process. Upon waiver approval, states usually have a set period of time to develop implementation protocols and an evaluation plan. The evaluation plans are based on the terms of the waiver as originally approved. However, since waivers were first approved, various aspects have been amended. For example, in response to legislative requirements, Arkansas added monthly contributions to its waiver and has indicated that it will seek additional amendments that could further change the program.11 Iowa’s initial plan included a Marketplace premium assistance waiver (for expansion adults above poverty); however, the Marketplace premium assistance program no longer is being implemented due to the loss of participating QHPs.

Looking Ahead

As additional states seek waivers to expand Medicaid or amend existing expansion waivers, it will be important to understand states’ experience with implementing current expansion waivers. A combination of federal and state waiver evaluations can capture cross-state assessment as well as in-depth state specific analysis to reveal which initiatives have been effective, which have proven too complicated, and which may add barriers to needed care for beneficiaries.  The combination of state and federal waivers (in addition to other independent research) can also help to guard against bias from a singular perspective (state or federal) given that the evaluations are taking place in an environment with a lot of scrutiny, consequences for beneficiaries, and often political stakes.  To learn from the expansion waivers, it will be important for the evaluations to be conducted and disseminated in a timely manner. Final results from the federal waiver evaluation will not be available until 2019, although interim reports will be available. However, states have already or will soon be required to release some waiver evaluation results. For example, Iowa already submitted data to CMS about the effects of its NEMT waiver, and Indiana is expected to release data from its NEMT waiver evaluation soon.12 Michigan has published analyses about reductions in uncompensated care after the expansion.13 Studies in Arkansas report on the early impact of its Marketplace premium assistance model on coverage, access to care, utilization, and other areas.14,15,16

For waiver evaluation results to be meaningful, it is also important for data and results to be accessible to policy makers, researchers, and the public in a timely manner. States are required to submit data to CMS quarterly, but much of this data is not public. Monitoring of waiver evaluations by stakeholders can help ensure that waiver implementation is on track and help policy makers to know if corrections are needed during the course of implementation. Keeping track of waiver evaluation results and understanding the issues and challenges that states and researchers faced in conducting these evaluations will help inform future efforts make changes to the Medicaid program.

Introduction Appendices

KFF Headquarters: 185 Berry St., Suite 2000, San Francisco, CA 94107 | Phone 650-854-9400
Washington Offices and Barbara Jordan Conference Center: 1330 G Street, NW, Washington, DC 20005 | Phone 202-347-5270

www.kff.org | Email Alerts: kff.org/email | facebook.com/KFF | twitter.com/kff

The independent source for health policy research, polling, and news, KFF is a nonprofit organization based in San Francisco, California.