The ALB performs health checks to the target group. If the target group uses HTTPS, the health check also uses HTTPS. The self-signed certificate on the EC2 instances is not trusted by the ALB, causing the health check to fail.
The ALB does not validate backend certificates by default, but the health check must succeed. The issue is that the health check path or port may be incorrect, but more likely the backend is not responding on the health check path. However, the most common issue is that the health check is not configured correctly.
Since the question states the health checks are failing, and the ALB returns 502, the likely cause is that the backend is not responding on the health check endpoint. But given the information, the self-signed certificate should work because ALB does not validate backend certificates. Actually, ALB can be configured to ignore certificate validation.
The default is to verify the certificate, but it can be disabled. So if the security team did not disable verification, the self-signed cert would cause failure. The correct answer is to disable certificate verification on the target group or use a trusted certificate.
The best action is to configure the target group to use protocol HTTP instead of HTTPS or disable certificate verification.