One of Australia’s largest electric vehicle charging networks, Chargefox, are pissed off at Australia’s largest telco, Telstra.
The issue began on Monday (3/6/24) around 2PM when Chargefox reported an ongoing Telstra service disruption which affected connectivity to a number of charging stations. This made it difficult (in some cases impossible) for users to access Chargefox charging services.
By 3:50PM, less than 2 hours later, the cause of the issue was determined to be a faulty storage array in Telstra’s Clayton data centre. Telstra described the issue as a major incident and yet here we are, more than 2 days later and the issue has not yet been resolved.
Chargefox is the customer we know about, however the issue is relates to a Telstra Cloud service for a number of Enterprise and Business customers.
Just before 5PM last night, Telstra issued an update to Chargefox about their impacted service.
Our team continues to work at pace on restoration efforts alongside our partner. While we identified the cause of the issue quickly, a faulty storage array, restoring service is taking longer than expected as we have identified new issues along the way.
They go on to say that it’s not as easy as simply restoring the data (I assume it was backed up), that the restoration tasks are lengthy, and each task must be planned, staged, analysed and monitored. Based on this, it sounds like its a bad day to be a Telstra engineer.
Further in the response, Telstra says they have made good progress towards the primary fix and now also have a number of customers back up and running.. ok, but Chargefox isn’t, so does that mean they were lower on the priority list?
Having enough electric vehicle infrastructure available is challenging at the best of times, but having one of the larger networks offline is quite frankly, pretty unacceptable and the blame for this is fairly and squarely on the shoulders of Telstra.
Today at 10am, Chargefox got their latest update from Telstra and it’s not good and you can quickly see their patients is running out, sharing the following on their incident page.
Chargefox has strongly and repeatedly requested an estimated time of resolution be provided by Telstra, especially in the face of this prolonged and unacceptable outage.
Telstra has to this point refused to provide even an estimated timeline for the current outage to be fixed.
We have been informed that this failure is impacting the communications of over 30,000 devices across Australia, including critical service providers.
Telstra continues to refer to ‘working with our partner’ which I assume is a cloud service provider like Microsoft or Amazon. Unless Telstra was using some kind of hybrid cloud arrangement, it doesn’t make sense that a disk array in their infrastructure would impact the cloud service provided to customers like Chargefox, anyway, I’m sure the post mortem will be extensive on this one.
Perhaps the most significant line from Telstra is this.
We know the time it is taking to restore your service and our inability to give you certainty on a restoration timeframe is clearly unacceptable.
When you openly and now publicly admit to your service not being acceptable, that’s a pretty good indication some money is about to be exchanged between the two parties for breaching Service Level Agreements.
The next update was due at 12.30 AEST, but almost an hour later, that hasn’t been shared by Chargefox.
For EV owners, Chargefox took the extraordinary step of recommending you seek alternative charging options (i.e. use the competition). Where possible, Chargefox also put their chargers into a mode that allows you to charge for free (they don’t have comms to know who charged and how much was taken).
What You Can Do
- Check the Chargefox app for real-time updates on station availability.
- Plan your trips accordingly and consider alternative charging options if necessary.
- Stay tuned to our social media channels for ongoing updates.