Identity Check Bias Evaluation
Identity Check uses facial recognition, a type of biometric technology. While biometric technology offers security and efficiency benefits, concerns have been raised around fairness or bias in the system. In response to these concerns, DIA commissioned the Identity Check bias evaluation in 2024.
As at September 2024 (just before the report was published) over 75,000 people had successfully used the Identity Check service across both the Ministry of Social Development and Hospitality New Zealand. Identity Check system data, held by DIA, showed a 94 percent success rate for users who completed the biometric processes.
The evaluation is just one step the Department is taking to ensure that our technology works fairly for everyone. DIA is committed to engaging and collaborating further with Māori and the wider community and intends to conduct further studies to ensure our systems are fair and fit for purpose.
Identity Check Bias Evaluation: Test Report
The independent biometric testing confirms that there is no evidence of significant bias in the Identity Check service, giving DIA and New Zealand public added confidence that the service works effectively for everyone.
BixeLab, an accredited biometric evaluation company, was commissioned to assess Identity Check for any potential bias. This is the first time a government biometric system has been tested for bias using a representative sample of New Zealanders. The results show no evidence of significant bias.
Previous testing illustrated relative performance across demographic groups. Even where differences existed, testing showed the algorithm performed well. This data was indicative only, which is why DIA commissioned New Zealand-specific analysis.
To date, biometric technology – currently used across a range of government services – has proven to be secure safe and compliant with all data protection requirements.
Read the full report: Identity Check Bias Evaluation Test Report
Report Summary
Key findings from the evaluation found:
- The system performed consistently and within acceptable limits for Māori, Pasifika, and non-Māori participants. No statistically significant disadvantage was found for any of these demographic groups.
- The matching engine showed reliable performance across all demographic groups for usability. Variation in the chance of false accept however was noted for Māori and Pasifika – investigations showed this was partly due to image quality issues.
- The system did not reject any genuine users based on incorrectly classifying a transaction as fraud.
The evaluation found that factors such as poor lighting or device quality can affect the capture of selfie images. Users with older hardware might have a higher likelihood of not meeting the minimum quality standards set by the app. The system interface provides guidance to users to help them capture quality images—such as ensuring good lighting and proper face positioning—to improve the success rate.
Evaluation Methodology
The evaluation of the Identity Check algorithm involved both front-end and back-end testing:
- Front-End Testing: A group of 148 volunteers submitted live selfie images using the Identity Check web application. This tested the system's ability to correctly detect and accept genuine users
- Back-End Testing: The system's matching engine was tested by comparing the selfie images against the stored passport photos, including both matches of the correct person and non-matches where the selfie and the passport photo were of different people.
Statistical analyses were then performed to understand whether the users were being disadvantaged based on their Māori or Pasifika heritage as compared to users of other decent.
The evaluation comprised 148 demographically diverse, volunteer test subjects recruited from within the Department.
The volunteers completed genuine testing using the DIA’s Identity Check web application. Forty-two subjects were of Māori heritage, 16 subjects were of Pasifika heritage, and 90 subjects were of other heritage. Each volunteer consented to participate in the evaluation, and their results were presented and recorded in a way that ensured that each test subject remains anonymous.
Next Steps
DIA will continue to monitor the performance of the Identity Check service and adjust and improve the technology to ensure the system is fair for all users.