Cybersecurity firm CrowdStrike has revealed that a bug in its quality control software caused the recent global IT outages. This bug allowed faulty data to be sent to millions of computers running Microsoft Windows.
CrowdStrike regularly sends out updates for its Falcon Sensor product, which helps protect computers against various threats and attacks. These updates come in two forms: Sensor Content and Rapid Response Content.
However, during a Rapid Response Content update on July 19, a broken file slipped through CrowdStrike’s quality control software and went unnoticed. This caused an out-of-bounds memory read error, resulting in a crash of the Windows operating system.
The incident review revealed that CrowdStrike relied on the Content Validator to catch any issues during Rapid Response Content updates. The assumption that these updates wouldn’t cause problems led to the problematic update being loaded into the Falcon Sensor.
To prevent similar incidents in the future, CrowdStrike plans to implement a staggered deployment strategy known as canary deployment. This involves initially rolling out updates to a small number of machines before proceeding with a global rollout.
The company also aims to enhance error handling in the Content Interpreter and introduce additional validation checks to the content validator. Furthermore, customers will now have the option to choose when and where these updates are deployed.
Following the IT outages, CrowdStrike’s stock value dropped by 20%, prompting the company to overhaul its content update process. CEO George Kurtz expressed his commitment to providing transparency regarding the incident and preventing similar occurrences in the future.