by Eric Dinallo, Avi Gesser, Erez Liebermann, Marshal Bozzo, Matt Kelly, Johanna Skrzypczyk, Corey Goldstein, Samuel J. Allaman, Michelle Huang, and Sharon Shaji

On January 17, 2024, the New York State Department of Financial Services (the “NYDFS”) issued a Proposed Insurance Circular Letter regarding the Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing (the “Proposed Circular” or “PCL”). The Proposed Circular is the latest regulatory development in artificial intelligence (“AI”) for insurers, following the final adoption of Colorado’s AI Governance and Risk Management Framework Regulation (“CO Governance Regulation”) and the proposed Colorado AI Quantitative Testing Regulation (the “CO Proposed Testing Regulation”), discussed here, and the National Association of Insurance Commissioners’ (“NAIC”) model bulletin on the “Use of Artificial Intelligence Systems by Insurers” (the “NAIC Model Bulletin”), discussed here. In the same way that NYDFS’s Part 500 Cybersecurity Regulation influenced standards for cybersecurity beyond New York State and beyond the financial sector, it is possible that the Proposed Circular will have a significant impact on the AI regulatory landscape.

The PCL builds on the NYDFS’s 2019 Insurance Circular Letter No. 1 (the “2019 Letter”) and includes some clarifying points on the 2019 Letter’s disclosure and transparency obligations. The 2019 Letter was limited to the use of external consumer data and information sources (“ECDIS”) for underwriting life insurance and focused on risks of unlawful discrimination that could result from the use of ECDIS and the need for consumer transparency. The Proposed Circular incorporates the general obligations from the 2019 Letter, adding more detailed requirements, expands the scope beyond life insurance, and adds significant governance and documentation requirements.

Legal Status of a NYDFS Circular Letter

The NYDFS’s circular letters do not change applicable law or regulations. The NYDFS uses circular letters to provide guidance to the insurance industry regarding how the NYDFS interprets existing law and regulations, to address issues and industry practices that it finds require changes and to clarify its expectations of the industry. The Proposed Circular, once finalized, would express the NYDFS’s guidance regarding existing laws (such as those prohibiting unfair discrimination) and regulations (such as those requiring a corporate governance framework, internal audit functions, and recordkeeping) in relation to AI and ECDIS. Circular letters often indicate the NYDFS’s enforcement priorities. Also, courts historically have given some deference to the NYDFS’s interpretations of laws that it is empowered to enforce (e.g., insurance, banking, and financial services laws) and its own regulations.

Key Elements of the Proposed Circular

Scope

The Proposed Circular outlines NYDFS’s expectations for all insurers authorized to write insurance in New York State (as well as licensed fraternal benefit societies, and the New York State Insurance Fund) that use ECDIS and/or artificial intelligence systems (“AIS”).

The PLC defines AIS as “any machine-based system designed to perform functions normally associated with human intelligence, such as reasoning, learning, and self-improvement, that is used – in whole or in part – to supplement traditional medical, property or casualty underwriting or pricing, as a proxy for traditional medical, property or casualty underwriting or pricing, or to establish ‘lifestyle indicators’ that may contribute to an underwriting or pricing assessment of an applicant for insurance coverage.”

Very similar wording is used to define ECDIS, which “includes data or information used – in whole or in part – to supplement traditional medical, property or casualty underwriting or pricing, as a proxy for traditional medical, property or casualty underwriting or pricing, or to establish ‘lifestyle indicators’ that may contribute to an underwriting or pricing assessment of an applicant for insurance coverage.” The Proposed Circular also provides that ECDIS “does not include an MIB Group, Inc. member information exchange service, a motor vehicle report, or a criminal history search.”

Unlike the CO Governance Regulation, the Proposed Circular’s definition of ECDIS does not include an enumerated list of factors that qualify as “lifestyle indicators,” such as social media habits, purchasing habits, home ownership, and educational attainment. Further, the scope of the PCL is broader than the CO Governance Regulation in that it applies to all types of insurance (rather than just life, at least for now, in Colorado), but narrower in that it applies to only underwriting and pricing, whereas the CO Governance Regulation applies to “other insurance practices,” which may include marketing and claims management.

However, the scope of the PCL is ambiguous in two significant areas. First, it is unclear whether uses of AIS that do not involve ECDIS are covered. The CO Governance Regulation is limited to the use of ECDIS, and one reading of the PCL is that it is similarly limited because of the parallel definitions of AIS and ECDIS. But an alternative reading is that the use of AIS is covered by the PCL, even if ECDIS is not involved, so long as AIS is used to “supplement traditional medical, property or casualty underwriting or pricing.” This is in an important scoping issue that should be resolved through the comment process.

Second, the language of the PCL suggests that it is limited to consumer insurance lines. For example, the PCL notes at the outset that the use of ECDIS and AIS “can both benefit insurers and consumers alike by simplifying and expediting insurance underwriting and pricing processes, and potentially result in more accurate underwriting and pricing of insurance. At the same time, ECDIS may reflect systemic biases and its use can reinforce and exacerbate inequality.” (emphasis added). However, when discussing the prohibition of unlawful discrimination in the underwriting process, the PCL also references New York Insurance Law § 2303, which applies to unfairly discriminatory rates for property and casualty insurance. Whether the PCL applies only to consumer insurance lines, or also applies to some forms of commercial insurance, is another important scoping issue that should be resolved through the comment process.

Fairness

According to the Proposed Circular, insurers are obligated under existing laws to establish that their use of ECDIS or AIS is not unfairly discriminatory, is supported by generally acceptable actuarial practices, is based on reasonably anticipated experience, and is not a proxy for any protected class. Insurers also bear the burden of ensuring that ECDIS or AIS provided by vendors complies with anti-discrimination laws, and insurers cannot rely solely on a vendor’s claim of non-discrimination.

Quantitative Testing

The Proposed Circular specifies that insurers should assess the use of ECDIS or AIS for any disproportional adverse effects in underwriting or pricing on similarly situated insureds or insureds of a protected class, which is notable as it goes beyond merely testing for effects on protected classes. Accordingly, the PCL should be read along with NYDFS Circular Letter No. 6 (2023), relating to “Unfair and Unlawful Discrimination in the Sale of Life Insurance and Annuities in the Individual Market and Certain Group Markets,” which discussed the NYDFS’s concerns regarding similarly situated consumers receiving different terms for the same policies.

The PCL provides a broad outline as to how insurers should test their ECDIS and AIS, stating that an insurer should not use ECDIS or AIS in underwriting or pricing unless it can establish through a comprehensive disparate impact assessment that the use does not unfairly or unlawfully discriminate. Such a quantitative assessment should, at minimum, include:

• Assessing whether the use of ECDIS or AIS produces disproportionate adverse effects in underwriting and/or pricing of similarly situated insureds, or insureds of a protected class.
  • If there is no prima facie showing of a disproportionate adverse effect, then the insurer may conclude its evaluation.
• If there is a prima facie showing of such disproportionate adverse effects, then the insured should assess whether there is a legitimate, lawful and fair explanation or rationale for the differential effect.
  • If no legitimate explanation can account for the differential effect, the insurer should modify its use of such ECDIS or AIS and evaluate the modified use.
• If a legitimate explanation can account for the differential effect, then the insured should conduct a search for a less discriminatory alternative variable or methodology that would reasonably meet the insurer’s legitimate business needs.
  • If a less discriminatory alternative exists, then the insurer should modify its use of ECDIS or AIS accordingly.

Compared to the CO Proposed Testing Regulation, the Proposed Circular offers much more flexibility to insurers in performing their quantitative assessments, and encourages insurers to use multiple statistical metrics in evaluating data and model outputs, including:

• Adverse Impact Ratio: Analyzing the rates of favorable outcomes between protected classes and control groups to identify any disparities.
• Denials Odds Ratios: Computing the odds of adverse decisions for protected classes compared to control groups.
• Marginal Effects: Assessing the effect of a marginal change in a predictive variable on the likelihood of unfavorable outcomes, particularly for members of protected classes.
• Standardized Mean Differences: Measuring the difference in average outcomes between protected classes and control groups.
• Z-tests and T-tests: Conducting statistical tests to ascertain whether differences in outcomes between protected classes and control groups are statistically significant.
• Drivers of Disparity: Identifying variables in AIS that cause differences in outcomes for protected classes relative to control groups. These drivers can be quantitatively computed or estimated using various methods, such as sensitivity analysis, Shapley values, regression coefficients, or other suitable explanatory techniques.

Qualitative Assessment

In addition to any quantitative analysis, insurers’ comprehensive assessment should involve a qualitative assessment of unfair or unlawful discrimination that includes being able to explain how the insurer’s AIS operates and the intuitive logical relationship between the ECDIS and an insured or potential insured individual’s risk.

Testing Frequency

The PCL provides that testing should be administered prior to putting AIS into production and on a regular cadence thereafter, as well as whenever material updates or changes are made to either the ECDIS or AIS.

Testing Documentation

The PCL provides that an insurer should document the processes and reasoning behind its testing methodologies and analysis for unfair or unlawful discrimination, and should be prepared to make such documentation available to the NYDFS upon request.

AI Risk and the Threshold for Quantitative Testing

The Proposed Circular notes that an insurer may deploy ECDIS and AIS in a variety of ways, and that the NYDFS recognizes that there is no one-size-fits-all approach to managing data and decisioning systems. Therefore, insurers “should take an approach to developing and managing their use of ECDIS and AIS that is reasonable and appropriate to each insurer’s business model and the overall complexity and materiality of the risks inherent in using ECDIS and AI.”

A risk-based approach should allow for some uses of ECDIS or AIS in underwriting or pricing to be deemed sufficiently low risk so as not to be subject to quantitative testing (or subject first to a qualitative assessment, and quantitative testing would only be required if the qualitative assessments yielded concerning results), but that possibility is not clear from the current draft of the PCL and is something that should be addressed in the comment period.

Testing Data Availability

The PLC provides that an insurer should not use ECDIS or AIS for underwriting or pricing purposes unless the insurer can establish that the data source or model, as applicable, does not use and is not based in any way on any class protected pursuant to the New York Insurance Law (“Insurance Law”) Article 26, which includes race, color, creed, national origin, disability, sex, marital status, being a victim of domestic violence, and past lawful travel.

Assuming those categories of protected classes are the same groups of individuals that are relevant for bias testing purposes, the Proposed Circular does not address how insurers should test for discriminatory impact on protected classes for which the insurer does not collect any data. For example, how would an insurer test to see whether its use of social media data is a proxy for national origin, or has a disparate impact on potential insureds who are victims of domestic violence, if it does not collect data that would indicate which insureds are part of those protected classes? Presumably, this data deficiency is the reason why the CO Proposed Testing Regulation limits its scope to assessing for discrimination on the basis of race, which can arguably be estimated using some combination of first name, last name, and location of home address. Indeed, the CO Proposed Testing Regulation specifically requires the use of Bayesian Improved First Name Surname Geocoding (“BIFSG”) to estimate certain categories of race of insureds for testing purposes. This gap between the testing obligations and the available data is another area that should be addressed in the comment process.

Transparency, Notice and Consumer Redress

For any adverse underwriting or pricing decision that was based on ECDIS or AIS, the insured must provide a notice to the insured or potential insured that discloses: (i) whether the insurer uses AIS in its underwriting or pricing process, (ii) whether the insurer uses data about the person obtained from external vendors, and (iii) that such person has the right to request information about the specific data that resulted in the underwriting or pricing decision, including contact information for making such request.

Insurers should also be prepared to respond to consumer complaints and inquiries about their use of AIS and ECDIS by implementing procedures to receive and address such complaints. Insurers must maintain any records of complaints regarding AIS or ECDIS and be prepared to make such records available to the NYDFS upon request.

Governance

According to the Proposed Circular, insurers are obligated to have a corporate governance framework that is appropriate for the nature, scale, and complexity of the insurer and provides appropriate oversight of the use of ECDIS and AIS.

Board and Senior Management Oversight

The PCL notes that the board of directors is obligated to oversee the insurer’s use of ECDIS and AIS and ensure that an effective governance framework is carried out. If certain board duties are delegated, as permitted, then the PCL provides that appropriate lines of reporting must be in place as well as regular, detailed reporting to the board. Further, the PCL explains that senior management should be responsible for the “day-to-day implementation” of ECDIS and AI systems, which includes:

• Establishing adequate policies and procedures;
• Assigning competent staff;
• Overseeing model risk management;
• Ensuring effective challenge and independent risk assessment;
• Reviewing internal audit findings; and
• Taking prompt remedial action when necessary.

Senior management should also ensure that all relevant operation areas are appropriately engaged, such as through a cross-functional management committee with representatives from key function areas, including legal, compliance, risk management, product development, underwriting, actuarial, and data science, as appropriate.

Auditing

The Proposed Circular expounds on insurers’ existing internal audit obligations under 11 NYCRR § 89.16. According to the PCL, insurers should ensure the internal audit function is appropriately engaged with the insurer’s use of ECDIS and AIS consistent with its financial, operational, and compliance risk. Such auditing should assess the overall effectiveness of the AIS and ECDIS risk management framework, which may include:

• Verifying that acceptable policies and procedures are in place and are appropriately adhered to;
• Verifying records of AIS use and validation to test whether validations are performed in a timely manner;
• Assessing the accuracy and completeness of AIS documentation;
• Evaluating the processes for establishing and monitoring internal controls;
• Assessing supporting operational systems and evaluating the accuracy, reliability, and integrity of ECDIS and other data used by AIS;
• Assessing potential biases in the ECDIS or other data that may result in unfair or unlawful discrimination against insureds or potential insureds; and
• Assessing whether there is sufficient reporting to the board or other governing body and senior management to evaluate whether management is operating within the insurer’s risk appetite and limits for model risk.

Policies and Procedures

The Proposed Circular provides that insurers using ECDIS or AIS should have written policies and procedures consistent with the Proposed Circular and which are approved annually by the board or senior management, including:

• Clearly defined roles and responsibilities;
• Monitoring and reporting requirements to senior management; and
• A training program for relevant personnel on the lawful use of ECDIS and AIS with an accountability mechanism for ensuring that all relevant personnel complete regular training in a timely manner.

Documentation

The Proposed Circular explains that insurers are obligated to maintain comprehensive documentation of all ECDIS and AIS use, whether developed internally or supplied by third parties. Insurers should be prepared to provide such documentation to the NYDFS upon request. The PCL indicates that comprehensive documentation may include:

• A description of the process for identifying and assessing risks associated with an insurer’s use of ECDIS and AIS;
• A description of associated internal controls designed to mitigate such identified risks;
• An up-to-date inventory of all AIS implemented for use, under development for implementation, or recently retired;
• A description of how each AIS operates, including any ECDIS or other inputs and their sources, the purpose and products for which the AIS is designed, actual or expected usage, any restrictions on use, and any potential risks and appropriate safeguards;
• A description of the process for tracking changes of an insurer’s use of ECDIS and AIS over time, including a documented explanation of any changes, associated rationale for such changes, and parties responsible for the approval of such changes;
• A description of the process for monitoring ECDIS and AIS usage and performance, including a list of any previous exceptions to policy and reporting;
• A description of testing conducted to periodically assess the output of AIS models, including drift that may result from the use of machine learning or other automated updates; and
• A description of data lifecycle management process, including ECDIS acquisition, storage, usage and sharing, archival, and destruction.

Risk Management

The Proposed Circular provides that insurers are permitted to manage the risks of AIS either within an existing enterprise risk management function, as required by the insurance law, or separately as part of an independent program. Regardless of an insurer’s choice, the PCL specifically provides that insurers should:

• Manage the relevant risks at each stage of the AIS life cycle and consider risk from individual AIS models and in the aggregate;
• Include standards for model development, implementation, use, and validation;
• Promote independent review and effective challenge to risk analysis, validation, testing, development, and other processes related to ECDIS and AIS; and
• Have competent and qualified personnel to execute and oversee AIS risk management with appropriate means of accountability.

Third-Party Vendors

The Proposed Circular provides that the obligation for compliance cannot be delegated to a vendor who is providing the ECDIS or the AIS, noting that insurers should:

• Retain responsibility for understanding any tools, EDCIS, or AIS used in underwriting and pricing for insurance that were developed by third-party vendors ensuring compliance with all applicable regulations;
• Develop written standards, for the use of ECDIS and AIS developed by a third-party vendor;
• Implement procedures for reporting any incorrect information to third-party vendors for further investigation and update, as necessary; and
• Develop procedures to remediate incorrect information from their AIS that the insurer has identified or has been reported to a third party.

Enforcement

The Department may audit and examine an insurer’s use of ECDIS and AIS, including within the scope of regular or targeted examinations pursuant to Insurance Law § 309, or a request for special report pursuant to Insurance Law § 308.

What steps can insurers take?

Insurance companies that would be subject to the guidance provided in the Proposed Circular should consider the following steps:

• Examine the requirements of the Proposed Circular and map the various stated obligations against their existing policies and procedures for use of ECDIS and AIS, in order to identify potential gaps. In particular, the testing requirements may require significant investments to operationalize.
• Assess whether any additional resources, in terms of personnel or budget will be needed in 2024 to meet the obligations set forth in the PCL.
• Determine whether any of the proposed requirements is too onerous or in conflict with typical business policies or other frameworks and, if so, consider contributing to comment letters. Note that the NYDFS was very receptive to comments during the Part 500 Cybersecurity rule-making process, so comments are very likely to be taken seriously.

Next Steps for the Circular Letter

DFS is requesting feedback on the proposed guidance by March 17, 2024. Comments should be submitted to innovation@dfs.ny.gov with the subject line “Proposed Circular on the use of AI and ECDIS in Insurance Underwriting and Pricing.”

Eric Dinallo, Avi Gesser, and Erez Liebermann are Partners, Marshal Bozzo, Matt Kelly, and Johanna Skrzypczyk are Counsel, Corey Goldstein, Samuel J. Allaman, and Michelle Huang are Associates, and Sharon Shaji is a Law Clerk at Debevoise & Plimpton LLP. The post was first published on the firm’s blog. 

The views, opinions and positions expressed within all posts are those of the author(s) alone and do not represent those of the Program on Corporate Compliance and Enforcement (PCCE) or of the New York University School of Law. PCCE makes no representations as to the accuracy, completeness and validity or any statements made on this site and will not be liable any errors, omissions or representations. The copyright of this content belongs to the author(s) and any liability with regards to infringement of intellectual property rights remains with the author(s).