If you’re on our security alert mailing list (and if not…let us know and we’ll add you!) you will have heard about the recent problems McAfee caused massive downtime to systems all across the world.
In trying to protect against a new threat they mistakenly turned on Windows itself!
While the damage wasn’t permanent any PC affected was unusable and couldn’t be repaired manually. We spent the back half of last week clearing the problem caused.
On Wednesday 25th April they released a virus definition file (5958 – April 21st) that incorrectly identified svchost.exe as a threat and deleted it on systems running Windows XP SP3.
Svchost is used for launching services (full description here) and any individual instance can run a group of services. This means its a pretty critical process!
Unfortunately for us a large chunk of our client base is running McAfee anti-virus software, the others run Trend Micro.
We knew something wasn’t quite right when we received several calls all around the same time with similar symptoms. However, while the symptoms were similar they weren’t identical so initially we didn’t quite know what was going on. Unfortunately the one thing they did have in common was a loss of network connectivity which meant we couldn’t fully diagnose the issue.
Later that day McAfee issued a notice, an updated definition file (5959) and details of how to fix the issue.
We had to,
Boot into safe mode
Add an EXTRA.dat to the c:\program files\commonfiles\mcafee\engine folder (or just run the 5959 Super DAT which is quicker)
Recover a copy of svchost from the service pack cache c:\windows\ServicePackFiles\i386\ or if not present, C:\WINDOWS\system32\dllcache\
Restart the computer
McAfee released an automated tool for this the following day (It’s in this KB article)
A simple enough fix to but as previously mentioned every PC we’d seen with this issue had no network connectivity.
This meant we potentially had to physically visit ever single PC we look after.
Only potentially because this only impacts running Windows XP SP3, incombination with a specific version of McAfee VirusScan we do have some clients running Vista or Windows 7. But most of our clients still currently run Windows XP. Also some of the PCs were still running 8.5.
Fortunately we also got a little lucky
We configure the McAfee products to fetch updates from the global McAfee update site every hour. Any servers on site will then check for and get updated every hour.
PCs check every 2-3 hours but we also put a random delay on this. The main reason is so that on larger sites we don’t want lots of PCs all generating network traffic at the same time. By putting in the random offset it’s staggered through the day. So this is in combination with the fact McAfee actually got the DAT update out the same day meant that lots of PCs never actually received the faulty update.
That said. We still had a LOT of work to do.
We visited as many sites as we physically could over a two day period and some other sites that had some tech savvy people on site we managed to go through it on the phone with them.
Obviously for our contract customers this was all at our expense.
McAfee have an FAQ here as well as a couple of blog post apologies.
As you can imagine there has been a lot of commentary on this and other vendors are jumping in to take advantage.
http://www.pcmag.com/article2/0,2817,2363018,00.asp
http://blogs.zdnet.com/Bott/?p=2031
By having a managed service contract in place many of our customer got this problem identified quickly and resolved fast.
| Next > |
|---|
