An older HP server running Windows 2003 Server Service Pack 2 presented today, the network interface was not functioning; it didn't matter what NIC was installed, this immediately pointed to the server software as the issue. The machine had an onboard Gigabit capable Intel Pro chipset. A firewall was disabled by someone else in case that was interfering, but it was not the issue. The machine was removed from site because reportedly networking had become 'intermittent' and then stopped, different ports were tried on the attatched switch and a different switch was also tried before the server was removed from site and brought into the workshop for further investigation.
I was able to assign a static IP address to the machine of 10.0.0.x / 255.255.255.0 with gateway 10.0.0.1. I knew these were valid settings on our network segment as I had other machines up on test, but was unable to ping the gateway from this particular server and recieved "Destination Unreachables". A similar story was evident with tracert (traceroute). "netstat -as" also revealed nothing particularly interesting. I also did a "route print" and flushed the routing tables, but nothing extraordinary was present there, arp -a came back with a big blank indicating the network interface was not fully initialised for use. I also performed a "netsh int ip reset resetlog.txt", disabled and re-enabled the network interface and applied the MAC address to the interface driver, all to no effect. A few people had already looked at the server and hadn't solved this issue yet.
I had a look, and weighed up some of the issues. I had thought perhaps an MTU or route issue initially. I had previously noted a service had been unable to start. In hindsight this was the first thing to actually look at. I Googled the symptom and found some mention of IPSec. Upon further investigation I checked services and found IPSec Services were down, when I tried to start IPSec it refused to start, and complained about a missing file. I had found the culprit service failing on boot. ..
The IPSec service does the following task under Windows:
"Manages IP security policy and starts the ISAKMP/Oakley (IKE) and the IP security driver."
With some further research I checked on the following registry key:
My Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\IPSec\
Policy\Local
or for the more initiated:
HKLM\SOFTWARE\Policies\Microsoft\Windows\IPSec\Policy\Local
What I found was rather interesting. The Windows and Windows NT keys just below the Policies registry key entry were in lowercase. eg. "windows" "windows nt", and didn't match the registry entries of a nearby Windows XP Pro machine which shares some networking similarities to Windows Server 2003. In addition to this the Entire IPSec key was missing. I manually re-created the IPSec key and IPSec\Policy, IPSec\Local, IPSec\Persistent keys that should normally be there.
The missing file that the IPSec Services appeared to be complaining about, was in fact the registry structure that had been somehow damaged. After recreating the keys/sub-keys the service started. Then the network interfaces worked once again!
IPSec is integral to the network interface functioning correctly as far as I can determine, although by all appearances tomost people it would seem like something you can live without. This server was running Routing and Remote Access also, providing VPNs.
It's likely the server refused to activate the interface until IPSec was capable of running, this _may_ be a security feature in Windows Server 2003 to prevent a server coming up without the ability to protect itself via IPSec policies/filters etc.
Total fault resolution time was around 25 minutes.
Comments