Section’s AI Proficiency Report: Key Findings and Takeaways
Executive Briefing: Measuring AI Proficiency in the Workforce – Key Takeaways and Insights
This briefing document analyzes the data and insights presented by Taylor Malmsheimer, COO & Head of Strategy at Section, during the “Measuring the AI Proficiency of the Workforce” Executive Briefing on October 23, 2024. The briefing primarily focuses on Section’s proprietary AI Proficiency Benchmark, a tool designed to assess and quantify AI proficiency in the workplace.
Core Assumptions & Methodology
The benchmark operates on the premise that AI proficiency directly correlates with productivity gains, leading to either cost savings or expanded capabilities. To measure proficiency, Section evaluates three key dimensions:
1. Usage: Self-reported data on AI usage frequency, platforms used, use cases, and perceived productivity gains.
2. Knowledge: Multiple-choice questions assessing understanding of AI capabilities, limitations, risks, and mitigation strategies.
3. Prompting: Free-response questions evaluating prompting skills across various scenarios, scored using AI.
Based on their combined scores across these dimensions, employees are categorized into five proficiency levels: AI Experts, AI Practitioners, AI Experimenters, AI Novices, and AI Skeptics.
Key Findings and Insights
Early Stages of Adoption:
- The majority of the workforce is still in the early stages of AI adoption, with most employees classified as AI Novices or Experimenters.
- “So most of the workforce is actually just getting started with AI. So the majority of employees here that you can see are actually AI novices in the kind of teal color here, and then AI experimenters in the blue color here.”
- Less than 10% of the workforce demonstrates true proficiency (AI Experts and Practitioners).
Industry & Functional Disparities:
- Language-intensive industries (e.g., Tech, Professional Services, Media) and functions (e.g., Marketing, Sales, Engineering) exhibit higher AI proficiency compared to more physical industries and functions (e.g., Manufacturing, Retail, HR).
- “So some of this variation isn’t super surprising, I would say, right? So functions that are more language-intensive, engineering, where you’re writing a lot of code, strategy, marketing, sales, those are the functions that we see are kind of further ahead in AI proficiency.”
Impact of Leadership and Management:
- Senior employees generally demonstrate higher AI proficiency than junior employees, likely due to prioritized rollouts for leadership and potential anxieties among junior staff regarding job security and proper AI usage.
- “So I see those more junior employees, they just might not have access to AI yet, right? Like when we talk to organizations, a lot of them are piloting with their managers or with their kind of more senior team. And that’s very fair, but it does mean that those more junior employees are gonna be more behind in access and proficiency.”
- Companies with explicit AI approval policies exhibit higher workforce proficiency compared to those with silent or restrictive stances.
- “So companies that explicitly approve of AI, probably not super surprisingly, have a more proficient AI workforce.”
- Individual manager attitudes significantly impact team proficiency, even within companies that encourage AI use.
Training Effectiveness and Shadow AI:
- Existing AI training primarily correlates with increased usage but not necessarily with improved knowledge or prompting skills.
- “But once we kind of dug in a bit more, AI training is only correlated with more usage of AI. It doesn’t actually seem to be right now correlated with better knowledge of how AI works, its risks, all those things, or with prompting.”
- Employees, even in companies with officially deployed LLMs, often utilize other AI tools, highlighting the prevalence of “Shadow AI” and the need for comprehensive data policies.
Actionable Recommendations
- Prioritize Language-Intensive Functions: Initiate AI rollouts in functions and industries where use cases are readily apparent and baseline proficiency is higher.
- Upskill HR Teams: Prioritize HR training to equip them to navigate AI deployment, address employee concerns, and promote responsible usage.
- Empower Junior Employees: Implement programs to address anxieties, provide access, and offer training tailored to the needs of junior staff.
- Establish Clear AI Policies: Communicate clear company stances on AI usage, promote responsible practices, and encourage open dialogue to mitigate “Shadow AI” risks.
- Train for Proficiency, Not Just Usage: Refocus training efforts to go beyond basic awareness and incorporate hands-on prompting practice, risk mitigation strategies, and use case identification.
Conclusion
This briefing underscores the critical need for organizations to shift their focus from mere AI adoption to cultivating true AI proficiency within their workforce. By addressing the highlighted insights and implementing the recommended actions, companies can unlock the full potential of AI and position themselves for success in an increasingly AI-driven world.