Rendered at 15:51:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lukaslueg 15 minutes ago [-]
Would be interesting if the report actually got into the details. The curiously specific "approx. 1ms window" sounds like an off-the-shelf memory race condition in combination with a preempting thread, for which 1ms is an absolute eternity and Guaranteed To Happen By Tuesday This Week™. But the "database corruption" sounds like database IDs being generated from timestamps with 1ms granularity.
macguillicuddy 37 minutes ago [-]
Interesting! Looks like a race condition (with a 1ms window) during squawk allocation (the process that gives each aircraft a 4 digit identifier for ATC purposes) caused data corruption. They present that window as being small but 1ms is a decent chunk of time in our work lol
crote 10 minutes ago [-]
More specifically: it sounds like something which was supposed to be an atomic operation was split into two by the preempting task, with corruption happening because that task also interacted with the same piece of (now-temporarily-invalid) data.
zrail 57 minutes ago [-]
Note: NATS the UK airspace manager, not NATS the pubsub system.
verst 13 minutes ago [-]
And here I was surprised they would use NATS in their implementation to assign squawk codes.
> The incident was triggered by a valid manual request for a squawk code. This manual request was made correctly and there was nothing abnormal or invalid about the associated flight plan.
> While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted.
> The reason this scenario has not occurred before is because:
> 1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond.
> 2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value.
> 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.
> Post-incident investigation has identified that when processing of the squawk allocation request resumed, the data associated with it had been corrupted and affected some subsequent flight data updates.
dtf 7 minutes ago [-]
> 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.
Well, that's comforting to know.
BBC: "Flight chaos caused by software defect in space of a millisecond, report says"
Sky: "'Millisecond' software error caused air traffic outage that grounded thousands of flights"
The Guardian: "Flight chaos for hundreds of thousands was caused in ‘millisecond’ by software error"
> While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted.
> The reason this scenario has not occurred before is because:
> 1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond.
> 2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value.
> 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally.
> Post-incident investigation has identified that when processing of the squawk allocation request resumed, the data associated with it had been corrupted and affected some subsequent flight data updates.
Well, that's comforting to know.
BBC: "Flight chaos caused by software defect in space of a millisecond, report says"
Sky: "'Millisecond' software error caused air traffic outage that grounded thousands of flights"
The Guardian: "Flight chaos for hundreds of thousands was caused in ‘millisecond’ by software error"
Sounds like pure bad luck.