Using Data to Advance Digital Skills: A State Playbook

To assist states in closing digital skill gaps and preparing for digital equity planning, this brief offers key questions and resources for state leaders to consider. This brief is a product of the NGA Workforce Innovation Network, an initiative of the NGA Workforce Development and Economic Policy Program.


(Download)

Digital skills – the skills associated with technology that enable users to find, evaluate, organize, create, and communicate information – are ubiquitous throughout the workforce. Yet, according to research by the National Skills Coalition, nearly one in three U.S. workers ages 16-64 have few or no digital skills, and at least 38 percent of those workers are employed in jobs that require moderate or advanced computer usage. Today’s changing labor market indicates that there is a critical need for action to ensure all workers gain the skills needed to meaningfully participate in work and life.

The Digital Equity Act (DEA), a provision within the Infrastructure Investment and Jobs Act (Public Law 117–58), includes a historic federal investment to promote digital equity, literacy, and inclusion. In particular, the State Digital Equity Capacity Grant Program will award nearly $1.5 billion in federal funds to states to support the development and implementation of Digital Equity Plans to promote digital equity and digital skill development. Governors have authority to select the entity that administers these plans and may select one of many eligible entities to implement the host of new grant programs that, together, can help states close the digital divide through promoting access, affordability, and adoption of digital technologies. To assist states in closing digital skill gaps and preparing for digital equity planning, this brief offers key questions and resources for state leaders to consider.

Disclaimer: In Spring 2022, states will have the opportunity to apply for federal Digital Equity Planning Grants funded through the U.S. Department of Commerce’s National Telecommunications and Information Administration. At the time of this publication, federal guidance on the data states will need to collect for this process has not yet been released. It is our hope that this publication can help state leaders get an early start on thinking through their vision and needs for digital skills data, so that they are prepared to hit the ground running as new federal requirements emerge. Once more information is available, this resource will be updated.

This Brief is Divided into Five Parts:

1) What Data do States Need to Collect?

Develop guiding questions: The creation of guiding questions can help state leaders refine a scope for understanding digital skill needs in their states, including their top priorities, populations, or programs and services for review. Such guiding questions can be used to inform data collection as well as help leaders understand the types of stakeholders that need to be engaged throughout the planning process. Sample questions for states include:

  • What information does our state need to inform our broader vision for developing digital skills among state residents?
  • Who are the people in “covered populations” as defined under Digital Equity Act Sec. 60302(8) (see Glossary), and how do we best measure digital skill assets and needs within each of those populations?
  • Where are the greatest gaps in digital skills programs including, but not limited to, foundational digital literacy training?
  • What are the skills workers need to meet the needs of the local labor market?
  • How can businesses play a role in helping to analyze data on digital skills needs in the workforce?

Determine what to measure: Once guiding questions are developed, the next step is to define key performance indicators that can help to answer the guiding questions, including:

  • Digital literacy: Define what it means to be digitally literate by applying existing definitions (see Glossary) for “workforce preparation” including digital skills and reviewing existing digital skill competencies frameworks. Digital skill definitions range from focusing on the discrete skills needed to navigate a computer or online environment to skills that are harder to measure, such as confidence and capacity to become lifelong learners of new technologies.
  • Digital skills in demand: This data can be gathered by tapping into states’ existing systems and processes for determining local employers’ workforce needs, such as state labor market information agency data; local workforce boards’ employer relationships; local chamber of commerce member surveys; and/or industry sector partnerships that bring together multiple employers along with an education or training provider.

Digital skills baseline data: Data on the existing digital skill levels of residents, particularly disaggregated by the “covered populations,” is difficult to obtain at the state and local level. States should consider reviewing available national data (see Commonly Used Datasets Matrix) and using those numbers as a starting point to apply to state residents. To more accurately “downscale” the national data, states should compare proportional differences in key demographic characteristics —  such as educational attainment, poverty, or limited English proficiency —  between the state’s population and that of the nation, using sources such as the Decennial Census and American Community Survey. Finally, states can look at their own administrative data (see Digital Skills Beyond the Workforce) to help them round out the picture of state residents’ digital skill abilities and needs.

2) How Can States Collect and Use Data?

Inventory existing datasets: States have a variety of data sources to inform their digital skills work, including education data, employment data, labor market information, public benefits data, and workforce program data. View the Commonly Used Datasets Matrix to learn more about how other datasets can be used to measure digital literacy and skill demands, and identify which covered populations are in most need of support.

  • Education data includes individual student records that schools and colleges, from preschool to postsecondary, keep to track the progress and development of students, including completion and types of degree or credential earned. This information can sometimes be made available in aggregate, non-identifiable form to other state agencies and stakeholders, such as a state higher education coordinating board.
  • Employment data is collected by state agencies, most often on a quarterly basis, to inform the state and the public about the employment status and earnings of residents in the state or region. Again, aggregate data is often made available to stakeholders through the state agency in charge of overseeing the Unemployment Insurance system or the state’s Labor Market Information agency.
  • Labor market information is the analysis of job markets by local areas, regions, or states based on real-time job postings of businesses and industries. Each state has its own labor market information agency, and private companies (i.e. Burning Glass or LinkedIn) also help states and cities make these real-time analyses of labor market data.
  • Public benefits data refers to information on individuals or families who receive public assistance, including unemployment insurance benefits, Temporary Assistance for Needy Families (TANF), or Supplemental Nutrition Assistance Program (SNAP). The data collected include the time and amount of assistance received and may also include data about education or workforce training programs in which they have participated.
  • Workforce program data includes records from state or federally funded workforce development programs that have reporting requirements, as well as programs sponsored through different industry associations, career technical education (CTE), apprenticeships, and adult education services. This includes programs funded through the federal Workforce Innovation and Opportunity Act (WIOA).

Identify what additional data needs to be collected: Once states have conducted an initial review of available data, it is important to identify gaps in the data and determine which gaps most need to be filled to develop effective Digital Equity Plans. States should distinguish between crucial data that is essential to have, and optional data that is helpful but not essential. Being as rigorous and specific as possible in documenting data gaps will help states make wise decisions in allocating their resources to fill these gaps.   For example, a state may identify a lack of data on rural residents’ digital skill needs and decide to prioritize data collection in rural regions by “over-sampling” survey respondents or partnering with additional rural organizations to gather data. As states prepare Digital Equity Plans, documenting gaps in data can provide valuable justification for implementation funds to support additional data collection efforts.

Develop a data collection plan: States should determine method(s) of data collection, who will be responsible for carrying out the plan, and what resources are available to implement. Most importantly, all data included in the data collection plan should directly contribute to answering one or more of the guiding questions discussed above. Including documentation of the link between each dataset and the question(s) it helps to answer can keep data collection efforts efficient and targeted, particularly with less structured qualitative methods. States should distinguish between analysis of existing datasets (which is cheaper but offers less precision) and new, original data collection (more expensive, but can be precisely targeted to the specific data a state needs). Data collection strategies include:

  • Analyzing and modifying existing data sources (listed above)
  • Collecting new quantitative data via surveys and/or direct assessments
  • Collecting new qualitative data via interviews and focus groups (which can add texture and context and validate findings from quantitative research)  

Leverage partners for data collection and analysis: Partnerships are crucial for enabling education and workforce systems to help individuals develop digital literacy and occupational skills. States can consider engaging a number of state and local partners based on what data needs to be collected. For example, states interested in gathering more data about the changing demands of skills needs from employers may wish to leverage partners at the state and local workforce development boards, institutes of higher education, and state departments of labor. Alternatively, states interested in learning about the gaps in foundational digital literacy may wish to leverage partnerships with adult basic education organizations and state and local libraries.

States should consider which of the following types of partners can best help them accomplish their goals:

  • Labor Market Information agencies, typically housed at state departments of labor
  • State workforce agencies operating reemployment programs (WIOA Title I), typically housed at state departments of labor
  • State and local Adult Basic Education programs (WIOA Title II)
  • State library associations and local public libraries
  • State and local workforce development boards
  • Institutions of higher education and state higher education coordinating boards
  • Governor appointed state infrastructure advisors/coordinators
  • Governors’ broadband leaders, including heads of State Broadband Offices 
  • Other divisions of government services (see Digital Skills Beyond the Workforce)

States may also consider partnering with organizations across the country that collectively comprise the “digital inclusion ecosystem,” such as the National Digital Inclusion Alliance and the Digital US Coalition led by World Education.

Establish baseline data: After layering data sets to better understand the digital skills of state residents and how these findings may differ from those identified in national data, states can develop baseline data against which to measure progress.

When developing baseline data, states should consider the following:

  • Identify ‘covered populations’ in states and communities, such as those listed in the Digital Equity Act (see Glossary)
  • Map existing assets
    • – Identify systems (e.g., public benefits, education, workforce development) and organizations (e.g., community-based organizations) currently providing digital equity services (including access, adoption, skill building, navigation) to covered populations
    • – Identify existing and pending digital equity initiatives (e.g., digital navigation, digital skill building, digital access) serving ‘covered populations’ which can help to collect or illuminate data
  • Establish measurable objectives in digital literacy to collect within each covered population, as required by the Digital Equity Act
3) Using Data for Storytelling

Turn data into stories: Data collected via the above-outlined process can help inform a state’s Digital Equity Plan, but can also form part of a larger aspirational vision for digital equity. States should capitalize on the rich array of data they are collecting and connect it to larger, existing stories, such as the story of a state’s goals for postsecondary attainment, a state’s journey toward closing racial equity gaps, or a state’s vision for connecting small and rural communities. For example, states may consider strategies to visualize gaps in digital equity and skill development using software, such as ArcGIS mapping, to highlight achievements or locations of policy focus.

Make data available for others’ storytelling: States should make every effort to provide aggregated, anonymized data in ways that nonprofit advocates and other stakeholders can use. By providing raw data via website or Application Programming Interface (API), states can make it easy for stakeholders to piggyback on the state’s prior work and use it to support shared goals and visions.

State Digital Equity Scorecard

The State Digital Equity Scorecard, created by the National Digital Inclusion Alliance (with support from Microsoft and the National Skills Coalition), offers states a public score based on the extent to which states are addressing digital equity across six key indicators:

  • data on digital skills needs
  • plan to close digital skill gaps
  • online digital skills training
  • incumbent worker training funds
  • technology apprenticeships
  • state broadband plans

States can use the interactive map to explore the methodology behind these factors, including links to original national and state datasets on state level job openings, openings requiring digital skills, and more.

4) Connecting Digital Skills Data to State Workforce and Education Goals

Incorporate findings into state plans: While states are putting in extensive effort to gather and analyze digital skills data, it is imperative to close the feedback loop and share that final data with workforce development, adult education, and higher education partners. States can leverage existing collaborative planning processes to ensure that findings are incorporated into regularly scheduled efforts such as WIOA plans, Perkins Career & Technical Education Act plans, etc.

Distribute findings to local recipients and service providers: States can leverage local partnerships to disseminate findings to local recipients of workforce and education funds, and to chambers of commerce, industry associations, and other business stakeholders. States may also consider partnering with municipalities to distribute local-level data and develop recurring feedback loops to inform state and regional goals.

5) Emerging Practices in Digital Skills Data Collection

Hawai‘i Digital Literacy and Readiness Study: Case Study

State of Hawai‘i Department of Labor & Industrial Relations Workforce Development

In 2021, the Hawai‘i Department of Labor and Industrial Relations (DLIR) sought to gather more information about the state of digital literacy and readiness among Hawai’i residents between ages 18 to 65 across the state. After approving definitions including “digital literacy” and “digital readiness,” the Workforce Development Council (WDC) launched an initiative to develop a statewide benchmark to quantify the digital readiness and literacy levels among residents of the State of Hawai‘i.

The objectives of the Digital Literacy and Readiness Study (DLRS) included:

  • Establishing a baseline measurement of the digital literacy and readiness of Hawaii‘s working age population;
  • Measuring the present rate of digital readiness statewide, disaggregated by demographics, education, occupation, industry, and geography;
  • Determining the proportion of adults who are not digitally literate; and
  • Identifying the largest population segments of adults who are not digitally ready.

The state measured literacy and readiness by demographics, education, occupation, industry, and geography and partnered with a private partner, Omnitrak, to collect individual-level survey data via landline and cell computer-assisted telephone interviewing and an online survey. As digital readiness has expanded beyond its initial hardware-oriented definition, self-efficacy, attitudes, and behaviors have become important dimensions in preparing residents for the digital age. This study analyzed underlying similarities and differences based on respondents’ self-assessment on seven self-reported aspects of readiness. Using cluster analysis, WDC identified five distinct groups of residents who fall along a continuum of digital readiness from relatively more ready to relatively unprepared based on geography, education, and other demographic features. 

Survey results found that Hawai‘i shows relatively lower rates of digital readiness compared to a national benchmark digital readiness survey conducted by the Pew Charitable Trust in 2015. With results from the study, the state proposed several recommendations, including a new Framework for Hawai’i Digital Literacy and Readiness. In partnership with employers, this Framework will help establish a baseline measurement of the current digital transformation stage of Hawai’i businesses and understand digital skills needed by the employers in key industries affected by digitization.

North Carolina’s Broadband Infrastructure Office of the Department of Information Technology, in partnership with the National Telecommunications and Information Administration, developed the  Digital Inclusion Plan Template and Guide to support North Carolina communities as they develop digital inclusion plans and digital equity strategies. This document includes both a template with the specific categories a community should consider including in their plan as well as a guide with probing questions to assist communities in the process of building that plan.

Oklahoma In an effort to gauge the role digital skills play in state occupational earnings and anticipated demand for those skills in the future, the Economic Research & Analysis Division of the Oklahoma Employment Security Commission recently conducted a study for the NGA Workforce Innovation state team. The Division used the digital skills scoring methodology from a 2017 Brookings Institution study and combined it with state data on job openings and wages produced by the division’s Bureau of Labor Statistics Occupational Employment & Wage Program and the Department of Labor Workforce Information Grant Projections. The study found a modest positive relationship between higher digital scores and average wage levels – offering the state new projections about how digital skills are related to occupational demand and economic outcomes, and new evidence about the importance of Oklahoma’s work to bridge the digital skills gap. State leaders plan to leverage this data and the NGA WIN grant to better prepare and inform the state’s strategic policy and programs to ensure a thriving, digitally skilled workforce.


Additional Resources:

Commonly Used Datasets to Measure Digital Literacy, Skill Demands, and Identify Priority Populations for Service

Data Source DescriptionObjectiveData ProviderIs state-level data available?Is data on digital skills available?Is data dis-aggregated by subpopulation?
U.S. Program for the International Assessment of Adult Competencies (PIAAC) Skills Map: a synthetic dataset that combines with U.S. Census data to create an interactive state and local map of adults’ literacy and numeracy skillsMeasure number of unemployed people and adults who are not in the labor market, as well as currently employed workers, with specific cognitive and workplace skillsProgramme for the International Assessment of Adult Competencies (PIAAC), administered in the United States by the U.S. Department of EducationYes; States may look to PIAAC’s Problem-Solving in Technology-Rich Environments (PS-TRE) data for national metrics on digital skillsYes (but only at the national level)Yes
Digital Nation Data ExplorerMeasure computer use, online activity, reasons for non-use, and devices used to access the InternetU.S. Census Bureau and NTIA YesNoYes
American Community Survey one- and five-year estimatesMeasure types of broadband technologies that households use to connect to the internet and the types of computers and internet devices they ownU.S. Census BureauYesNoYes
National Adult Education Reporting SystemMeasure basic education levels for individuals enrolled in Adult Basic Education programsU.S. Department of EducationYesNoYes
Occupational Information Network (O*Net)Measure skills, abilities, knowledge, work activities, and interests associated with occupations, over timeU.S. Department of LaborYesYesNo
Data from assessment providersMeasure individual mastery of basic digital skillsExample: Northstar Digital Literacy AssessmentPotentially, depending on the provider; currently states must aggregate data collected locallyYesYes, potentially
Private research on changing employer demand for digital skills as measured by job ads and related dataMeasure changing employer demand for digital skillsPrivate research companies such as Burning Glass, Accenture, or LinkedInYesYesNo
Credential attainment Measure number of students earning a credential with specific skills, abilities, knowledge, work activitiesState institutions of higher educationYesYesUnknown

The Role of Supplemental Skills Assessments

Administering formal assessments can be a resource intensive yet valuable exercise to gather more local-level data about populations in need of skills training. Assessments can be conducted either at the community or individual level, and results can be aggregated to help guide the training and services offered.

Digital skills assessments come in various forms and are designed for different purposes. A recent landscape scan by the federally funded Digital Resilience in the American Workforce (DRAW) initiative found that assessments range from self-assessments and inventories, performance- or competency-based assessments, portfolios, and more standardized assessments. Assessments also vary as to the skills they measure, depending on the skills definitions or frameworks to which they were aligned. Digital skills needs change rapidly, challenging assessment providers to keep up. Given the need to be able to adapt to new software and technologies, it is important to consider whether assessments measure performance on a variety of software interfaces, not just in one environment.

This is especially critical for reasons of equity as individuals and communities use technology differently; for example, many immigrant populations were using mobile technologies in effective ways even before assessments based on computer-based digital literacy measured that.

DRAW research identified that the Northstar Digital Literacy Assessment is currently the most used across adult education. A program of Literacy Minnesota, Northstar defines, assesses, and teaches basic skills needed to use computers and the internet in daily life, employment, and higher education. It was originally designed as a pre-training diagnostic, so its assessment data can be used to inform instruction and accelerate learning for individuals and groups. States have since used data collected to determine what digital skills learners already know and to collaborate with employers and educational institutions to screen residents into work and learning opportunities.

To learn more about assessments, the International Telecommunications Union offers a comprehensive guidebook and analysis of national digital skills assessments.


Digital Skills Beyond the Workforce

While much of the digital skill-building conversion centers on working-age adults who are employed or looking for work, the Digital Equity Act requires states to consider a broader array of state residents in developing and implementing their Digital Equity Plans. People of all ages need digital skills to pursue education, navigate banking and financial services, obtain telehealth care, access public services, and participate fully in social, civic, and economic life.

States can gather information on this broader population using many of the same sources discussed in the “Type of Data” section of this document. In addition, states should consider creative use of administrative data to shed light on digital skills, for example:

  • Analyze data on state Unemployment Insurance claimants to see which geographic or demographic factors are correlated with people who contact the UI office by phone versus using the internet. This data could help illuminate particular zip codes, age groups, or industries that have lower percentages of UI claimants using the internet to access public services. In turn, this could identify areas with poor broadband access, limited ownership of digital devices, limited digital skills, or other digital equity barriers.
  • Review public benefits web form-submission data to determine how many TANF or SNAP applicants are applying for benefits using a mobile device rather than a desktop or laptop computer. This data can help identify particular geographic or demographic groups that may rely on smartphone-only internet access and thus are more likely to lack the digital devices and broadband access necessary to participate fully in online learning.
  • Explore opportunities to review aggregate, anonymized data from the state higher education system to identify groups of students that have not accessed online university services, or have accessed such services only from a mobile device. Discovering this information can help states to identify broad categories of students who may face barriers to digital inclusion.
  • Review aggregate, anonymized data from the K-12 public education system to identify the percentage of students participating and measuring proficiency in computer science courses and technology training that leads to an industry-recognized credential. Consider how to aggregate data by sub-population to identify gaps in participation for those traditionally underserved, such as women and students of color.

These are just a few examples of how state agencies can review existing state data in ways that do not violate the privacy or security of individual state residents. Once initial issues are identified, states can consider using best practices from the “Civic Tech” field to conduct user surveys or studies to more thoroughly explore the reasons behind initial findings. For example, this blog post from the California Office of Digital Innovation explains how the state conducted 60 informal interviews with state residents who were struggling to access federal Emergency Broadband Benefits.


Glossary

These definitions are taken directly from the Digital Equity Actas passed by Congress on November 15, 2021, as part of Public Law 117–58:

  • Digital Equity: the condition in which individuals and communities have the information technology capacity that is needed for full participation in the society and economy of the United States.
  • Digital Inclusion: the activities that are necessary to ensure that all individuals in the United States have access to, and the use of, affordable information and communication technologies, such as i) reliable fixed and wireless broadband internet service; ii) internet-enabled devices that meet the needs of the user; and iii) applications and online content designed to enable and encourage self-sufficiency, participation, and collaboration; and includes i) obtaining access to digital literacy training; (ii) the provision of quality technical support; and (iii) obtaining basic awareness of measures to ensure online privacy and cybersecurity.
  • Digital Literacy: the skills associated with using technology to enable users to find, evaluate, organize, create, and communicate information.
  • Covered Populations: those living in low-income households or rural areas, older adults, veterans, racial or ethnic minorities, and individuals with disabilities, language barriers, or who are incarcerated in non-federal institutions.

Resource Links

Assessment Resources

Mapping the Digital Skills Divide

Additional Reading


Acknowledgements

This brief was prepared by Katherine Ash, Consulting Director of the NGA Workforce Innovation Network; Amanda Bergson-Shilcock, Senior Fellow, National Skills Coalition; Alison Webber, Director of Strategic Initiatives at EdTech Center @ World Education; Judy Mortrude, Senior Advisor, World Education; and Aaron Schill, Director of Strategic Initiatives at the National Digital Inclusion Alliance.

The authors wish to thank the states participating in the NGA Workforce Innovation Network, which provided valuable insights and expertise that contributed to this resource. This brief was made possible with support from the Cognizant Foundation, the founding sponsor of the NGA Workforce Innovation Network, with additional support from Intel, Microsoft, Walmart, and Western Governors University. We thank them for their support and acknowledge that the findings, conclusions, and recommendations presented in this report are those of the authors alone, and do not necessarily reflect the opinions of the sponsors.

National Digital Inclusion Alliance (NDIA) is a US nonprofit organization that brings together over 600 non-profit organizations, policy-makers, and academics to advocate for national access to broadband and to put an end to the multiple digital divides.

National Governors Association Center (NGA Center) for Best Practices teams develop innovative solutions to today’s most pressing public policy challenges. The center is the only research and development firm that directly serves the nation’s Governors. The Workforce Innovation Network is an initiative of the Workforce Development and Economic Policy Program.

National Skills Coalition (NSC) is a national organization fighting for inclusive, high-quality skills training so that more people have access to a better life, and more local businesses see sustained growth. Since 2000, through expert analysis and technical assistance, broad-based organizing, targeted advocacy, and cutting-edge communications, NSC has changed hundreds of state and federal skills policies that have changed thousands of lives and grown local businesses and economies.

World Education-US advances equity through education and improving the availability, quality, and capacity of adult education & workforce programs. Its award-winning EdTech Center advances digital equity and supports organizations to leverage technology to accelerate impact.