Telstra’s only two timing experts had been sent home to rest before network fell over

The only two Telstra engineers with deep expertise in the system that failed in July had been sent home on mandatory rest by the time the national mobile outage they had unwittingly triggered took hold, an external investigation has found.

The outage cut hundreds of people off from Triple Zero , halted trains in Victoria and NSW and brought down eftpos systems across the country. It has triggered a Senate inquiry and an investigation by the communications regulator that could see Telstra fined up to $30 million.

The pair had replaced a faulty power supply in a timing chassis in Melbourne in the early hours of July 8. When it came back online at 2.50am, a GPS card inside reset and began telling the mobile network the date was 2006. The card was missing a firmware update for a predictable fault the supplier had already flagged in bulletins, confirming this masthead’s July reporting .

The report by Technology Audit Partners found alarms on the timing servers were checked only during business hours by a small number of staff, and did not appear in the monitoring tools used by round-the-clock support teams. When things went wrong, nobody on duty could find records of the servers, the work done that night or the people to call.

The outages have heaped fresh scrutiny on Australia’s Triple Zero system and the reliability of the nation’s telecommunications networks more broadly.

“At its core, this came down to the network timing system not being given the priority that a critical network capability requires,” she said on Wednesday. “That is a miss on our side.”

Brady also revealed Telstra had missed eight failed Triple Zero calls during the outage and never carried out the welfare checks required to find out whether the people who made them were safe. She blamed a manual processing fault and a monitoring fault.

The checks have since been done. One caller got through on another network, and another found a different way to connect. Five either needed no help or had misdialled. Police visited the home of the unidentified eighth caller.

“It shouldn’t have happened. We should have picked those eight up,” Brady said. “We know how important Triple Zero is.”

The eight are on top of the hundreds of unsuccessful Triple Zero calls Telstra has already disclosed.

The telco also revised the start of the outage to 2.50am on July 8, almost 50 minutes earlier than it had said. The previous time reflected when a maintenance ticket was closed rather than when the fault began.

Brady said the failure was “not due to infrastructure failing”, but to an undocumented design change made in October last year and a software update that was never applied to a GPS card.

Despite the auditors finding the team responsible had been constrained by budget priorities, Brady rejected suggestions of underinvestment, saying Telstra employed about 2800 network engineers and had spent more than $19 billion on capital works and spectrum over five years.

“It’s not about not enough resources. It’s not about not enough money,” she said. “It’s about the prioritisation.”