California Issues Guidelines for State Agencies Purchasing Generative AI Tools
April 2024
Number 18
Building upon the report by the California Government Operations Agency, entitled State of California: Benefits and Risks of Generative Artificial Intelligence Report and published in November 2023, California has issued guidelines addressing procurement, use, and training related to generative artificial intelligence (GenAI) by State agencies. Authored by the California Department of Technology, Department of General Services, Office of Data and Innovation, and Department of Human Resource, the guidelines set out best practices intended to support agencies in implementing and overseeing the ethical, transparent, and trustworthy use of GenAI. Although these guidelines do not apply to school districts or other local agencies at this time, the concepts and limitations included in them may be a precursor to future GenAI guidelines, orders, and/or legislation.
State Agency Requirements
At the outset, the guidelines emphasize department accountability, warning that “State entities and their respective leadership will ultimately be responsible for evaluating and incorporating GenAI to support each entity’s unique structure and mission.”
As part of ensuring accountability, the guidelines specify agency responsibilities related to both incidental and intentional purchases of GenAI.
Incidental GenAI Purchases
For incidental GenAI purchases (purchases where the State or vendor identifies a subcomponent of the purchase as using GenAI tool(s) to assist with the delivery of the solution), State agencies must: (1) assign an executive-level team member responsible for continuous GenAI monitoring and evaluation; (2) attend mandatory executive and procurement team GenAI trainings: and (3) review annual employee training and policy to ensure staff understand and acknowledge acceptable use of GenAI tools.
Intentional GenAI Purchases
For intentional GenAI purchases (purchases for which a State entity identifies a GenAI product or solution to meet a business need for any type of procurement), State agencies must also: (1) identify business needs and understand the implications of use; (2) create a culture of engagement and communication between State staff and employee end users; (3) assess the risks and impacts of deploying the GenAI; (4) prepare data inputs and test models prior to deployment; and (5) establish a GenAI-focused team responsible for the continuous evaluation of GenAI use and its implications.
All State entities will also be required to conduct and submit an inventory of all uses of GenAI, both intentional and incidental, to the California Department of Technology. Instructions for these submissions have not yet been released.
Further, State agencies will be required to complete a GenAI risk assessment based on the National Institute of Standards and Technology’s (NIST’s) “AI Risk Management Framework” and relevant portions of the “State Administrative Manual” and “State Information Management Manual.”
Takeaways
State agencies are ultimately responsible for ensuring appropriate agency use of GenAI. To that end, State agencies and departments are required to designate employees responsible for monitoring GenAI tools and assessing their risk, modify their procurement practices as outlined in new State guidelines, and ensure employees are properly trained in GenAI use.
The California Department of General Services and California Department of Technology will update guidance as the implementation of GenAI is tested and analyzed. The current guidelines will serve as interim guidance until California publishes its final procurement and training policy in 2025.
Lozano Smith is committed to remaining at the forefront of addressing legal issues related to artificial intelligence. If you have questions about your agency’s obligations as they relate to artificial intelligence, please contact the authors of this Client News Brief or any attorney at one of our eight offices located statewide. You can also subscribe to our podcasts, follow us on Facebook, Twitter and LinkedIn or download our mobile app.
Number 18
Building upon the report by the California Government Operations Agency, entitled State of California: Benefits and Risks of Generative Artificial Intelligence Report and published in November 2023, California has issued guidelines addressing procurement, use, and training related to generative artificial intelligence (GenAI) by State agencies. Authored by the California Department of Technology, Department of General Services, Office of Data and Innovation, and Department of Human Resource, the guidelines set out best practices intended to support agencies in implementing and overseeing the ethical, transparent, and trustworthy use of GenAI. Although these guidelines do not apply to school districts or other local agencies at this time, the concepts and limitations included in them may be a precursor to future GenAI guidelines, orders, and/or legislation.
State Agency Requirements
At the outset, the guidelines emphasize department accountability, warning that “State entities and their respective leadership will ultimately be responsible for evaluating and incorporating GenAI to support each entity’s unique structure and mission.”
As part of ensuring accountability, the guidelines specify agency responsibilities related to both incidental and intentional purchases of GenAI.
Incidental GenAI Purchases
For incidental GenAI purchases (purchases where the State or vendor identifies a subcomponent of the purchase as using GenAI tool(s) to assist with the delivery of the solution), State agencies must: (1) assign an executive-level team member responsible for continuous GenAI monitoring and evaluation; (2) attend mandatory executive and procurement team GenAI trainings: and (3) review annual employee training and policy to ensure staff understand and acknowledge acceptable use of GenAI tools.
Intentional GenAI Purchases
For intentional GenAI purchases (purchases for which a State entity identifies a GenAI product or solution to meet a business need for any type of procurement), State agencies must also: (1) identify business needs and understand the implications of use; (2) create a culture of engagement and communication between State staff and employee end users; (3) assess the risks and impacts of deploying the GenAI; (4) prepare data inputs and test models prior to deployment; and (5) establish a GenAI-focused team responsible for the continuous evaluation of GenAI use and its implications.
All State entities will also be required to conduct and submit an inventory of all uses of GenAI, both intentional and incidental, to the California Department of Technology. Instructions for these submissions have not yet been released.
Further, State agencies will be required to complete a GenAI risk assessment based on the National Institute of Standards and Technology’s (NIST’s) “AI Risk Management Framework” and relevant portions of the “State Administrative Manual” and “State Information Management Manual.”
Takeaways
State agencies are ultimately responsible for ensuring appropriate agency use of GenAI. To that end, State agencies and departments are required to designate employees responsible for monitoring GenAI tools and assessing their risk, modify their procurement practices as outlined in new State guidelines, and ensure employees are properly trained in GenAI use.
The California Department of General Services and California Department of Technology will update guidance as the implementation of GenAI is tested and analyzed. The current guidelines will serve as interim guidance until California publishes its final procurement and training policy in 2025.
Lozano Smith is committed to remaining at the forefront of addressing legal issues related to artificial intelligence. If you have questions about your agency’s obligations as they relate to artificial intelligence, please contact the authors of this Client News Brief or any attorney at one of our eight offices located statewide. You can also subscribe to our podcasts, follow us on Facebook, Twitter and LinkedIn or download our mobile app.
As the information contained herein is necessarily general, its application to a particular set of facts and circumstances may vary. For this reason, this News Brief does not constitute legal advice. We recommend that you consult with your counsel prior to acting on the information contained herein.
