azureblog.co.uk
← cd ~/posts

Test Conditional Access safely with report-only mode and What If

Most Conditional Access outages come from a policy that did something nobody expected. Two built-in tools let you see exactly what a policy would do before it does it.

2 min read⚠ checked 26 Nov 2025Entra ID · Conditional Access
Part 2 of 7 in Conditional Access from zero
On this page
  1. Report-only mode
  2. Reviewing the impact
  3. The What If tool
  4. Reading report-only results
  5. A safe rollout pattern

is powerful precisely because it can block anyone from anything. That's also why a careless change can lock out a whole organisation. Two tools take most of the risk away.

  1. 01Create policy in report-only
  2. 02Collect a week of sign-ins
  3. 03Review reportOnlyFailure results
  4. 04Pilot group
  5. 05Enforce for everyone

Report-only mode

Set a policy's state to Report-only and Entra evaluates it on every sign-in without enforcing it. The result is recorded in the sign-in logs on the Report-only tab: would it have applied, and would the user have passed or failed?

Run new policies in for at least a week, so you see a full working cycle including remote work, mobile devices and service accounts.

Off

  • Not evaluated at all
  • Safe place to draft a policy

Report-only

  • Evaluated on every sign-in
  • Result logged, never enforced
  • Use for at least a week

On

  • Enforced
  • Users can be blocked or prompted
  • Keep break-glass excluded
The three policy states.

Reviewing the impact

The Conditional Access insights and reporting workbook summarises report-only results across all sign-ins. With logs in , you can also query it directly:

kql
SigninLogs
| where TimeGenerated > ago(7d)
| mv-expand ConditionalAccessPolicies
| where ConditionalAccessPolicies.displayName == "Require compliant device"
| where ConditionalAccessPolicies.result == "reportOnlyFailure"
| summarize Failures = count() by UserPrincipalName, AppDisplayName
| order by Failures desc

Every reportOnlyFailure is someone who would have been blocked. Work through the list before you switch the policy on.

The What If tool

answers a specific question: if this user signed in to this app, from this platform and location, which policies would apply? You'll find it on the Conditional Access Policies page. It's ideal for troubleshooting a single user's ticket and for checking your exclusions are doing what you expect.

Reading report-only results

ResultWhat it means
reportOnlySuccessThe user would have met the policy's requirements
reportOnlyFailureThe user would have been blocked. Investigate before enforcing
reportOnlyInterruptedThe user would have been prompted, for example for
reportOnlyNotAppliedThe policy's conditions didn't match this sign-in

One known quirk: Microsoft notes that report-only policies requiring a compliant device can still prompt users on macOS, iOS and Android to pick a device certificate. Test on those platforms before a wide report-only rollout.

A safe rollout pattern

  1. Create the policy in report-only.
  2. Review the results after a week and fix the gaps.
  3. Enable it for a pilot group, then everyone.
  4. Keep excluded throughout.
  1. Policy created in report-only
  2. Review failures, fix gaps
  3. On for pilot group
  4. On for everyone
A typical timeline for a new policy.

Try the sign-in log explainer when reviewing individual sign-ins: paste the exported JSON to see which policies applied and why.

This post was last checked against Microsoft's documentation over six months ago. The approach should still hold, but check the linked sources for anything that has changed before you act on it.

Next in Conditional Access from zero · part 3 of 7Authentication strengths: requiring the right kind of MFA →

Related posts