Autopilot Failures - Mystery Solved
Recently, I was notified about an odd issue where an Autopilot build started failing. The build had not changed for months but started failing with a very odd error in the device preparation phase.
(Apologies for the very bad photo of a very dirty test laptop)
Investigating the Fault
My initial thoughts were that this was a Microsoft code fault with the latest service release (the issue happened concurrently with the roll-out of the 2009 service release for Intune).
The affected organisation logged a support call and I provided some guidance on resolution.
The response from Microsoft Support was:
In the log shared with us, we can see
[HKEY_LOCAL_MACHINE\software\microsoft\windows\autopilot\EnrollmentStatusTracking\Device\DevicePreparation\PolicyProviders\Sidecar]
"InstallationState"=DWORD:00000001
In order to pass ESP, this value should be either 2 or 3 (2= not required, 3 = completed)
https://docs.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp
So now we had something to investigate in depth. We started looking at the behaviour of the Intune Management Extension installation during enrolment. Two things jumped out.
The Intune Management Extension was successfully installed each time
Win32 apps and PowerShell scripts were being run (at least some of them)
The InstallationState value was consistently not being updated to 3 (meaning completed) so the Enrolment Status Page consistently recorded a failure even though the enrolment had succeeded.
Root Cause
Akshay Patel started taking the build apart and removing policies, apps and scripts to determine whether something in the build caused the failure. An errant script that triggered BitLocker encryption was identified as the culprit. If the script was removed then Autopilot enrolment succeeded.
The BitLocker script was something I wrote in 15 minutes, on-site at a customer, back in early 2019. It was a quick and dirty fix to ensure that all devices had an encrypted OS drive if the automatic encryption did not work (happens surprisingly often). I knew the script had problems and had always intended to re-write the script but priorities were elsewhere.
The real mystery was why this script was suddenly a problem when it had worked for months in this deployment.
My working theory is that the Intune Management Extension (Sidecar) was updated in September 2020. Subsequently any PowerShell script that generated an unhandled error during execution became blocking and stopped the Intune Management Extension installation reporting a successful installation. The PowerShell script had a loop that did not exit cleanly if an error was encountered, which did not help.
Removing the errant BitLocker script resolved the issue but the change in behaviour is a concern.
Lessons Learnt
I (re)learnt the following lessons from this incident:
PowerShell scripts deployed via Intune/Microsoft Endpoint Manager need more care!
Stuff that you know is broken should be fixed!
The Intune Management Extension is still a relatively fragile component
I use a few PowerShell scripts for Autopilot deployments. The scripts are dirty hacks for configuration changes that were not available in CSP policy at the time the scripts were written. Going forward, I am going to look harder at moving those setting into CSP policy. I am also going to look at a better mechanism for deploying a standard device configuration script for Autopilot devices (more later on that subject).
BitLocker Script
I rewrote the BitLocker script to make the script less bad. The script errors less often and no longer blocks Autopilot enrolments. A better mechanism for handling the exit to a Do Until loop and a bunch of other improvements were added.
The script is definitely not perfect but will suffice until I get time to work on a more permanent solution to the problem of devices that do not encrypt automatically.
You can download the BitLocker script from here [LINK]. You may find the script useful for any deployments with devices that do not automatically encrypt.
Acknowledgements
Thank you to Akshay Patel for bringing this issue to my attention and spending hours on the phone to Microsoft support.
Thank you also to Intune Support, who I reached out to for help on Twitter, and provided some useful feedback.