February’s Patch Tuesday was eventful to say the least. Many have noticed that Patch #977165 (Security Bulletin MS10-015) causes a blue screen on some systems (Stop Error: PAGE_FAULT_IN_NONPAGED_AREA). While the initial outrage was directed at Microsoft for a shoddy patch, eventually it was found that the Alureon Rootkit was the cause of the blue screen after KB977165 was installed.

But don’t worry! – the makers of the Alureon Rootkit have actually updated it and patched the flaw! Hurray!

For everyone else there are two options:

1. Use a LiveCD to scan your hard drive for the rootkit and remove it. This will resolve the issue. Try Knoppix STD (http://www.knoppix-std.org/) or BartPE (http://www.nu2.nu/pebuilder/).
2. Remove MS10-015 (977165) from your system.

How to remove Security Bulletin MS10-015 (977165) from your system

1. Insert the Windows XP disc into the CDROM. You may need to adjust your BIOS settings to boot the CDROM first.
2. When the “Welcome to Setup” screen appears, press “R.”
3. Select the installation you wish to access (there should be only one option for most systems).
4. Enter the administrator password when asked.
5. Once at the Recovery Prompt, press ENTER after typing the following command: CHDIR $NtUninstallKB977165$\spuninst
6. Press ENTER after typing the following:  BATCH spuninst.txt
7. Press ENTER after typing the following:  systemroot
8. Press ENTER after typing the following:  exit
9. Remove the Windows XP CD and restart.

No Comments |  Print This Post |

Backups are a necessary evil in System Administration, and although most of us dislike the process, it is by far the most critical element under the IT umbrella. I like to think of the whole process as a Recovery Plan instead of a backup plan, because in the end all I care about is that the data is recovered properly and quickly. One of the biggest pitfalls new System Administrators or System Administrators new to a particular company do, is that they do not test their own backups. Not being able to recover information from a system you designed or recommended is the quickest and surest way to get fired. 

 

1. Risk Assessment

Although this sounds simple, a true risk assessment is rarely done and far out of the reach of the average business. Although hiring actuaries and combing through insurance statistics is ideal, this is far from what companies are willing to do for a data recovery plan. Many System Administrators find companies that have not experienced data loss are less willing to be thorough in their analysis and budgeting.

One of the first things in a recovery plan is to write down the possible external risks, some examples:

1. Fire
2. Flood
3. Earthquake
4. Tornado
5. Physical Break-in / Theft
6. Virtual Break-in (Cracker)

Ask your Insurance Company what some likely issues will be, they will be happy to tell you every possible disaster scenario.

Next, think of internal risks such as:

1. Viruses/Malware
2. Data Corruption
3. Hardware Failure (Electrical or Mechanical)
4. User Error (accidental deletion)
5. User Malice (non-accidental deletion)

To supplement risks, look through your company’s history for any previous data loss and the reasons for it. Enlist the help of your industry colleagues for any scenarios not on the list. You’ll be amazed at some of the “once-in-a-lifetime” stories you’ll hear – some may be applicable to you.

 

2. Impact Rating

An impact analysis on each scenario listed above should be created. This involves the hardware, software and data (all systems) that are affected and how. Does the impact involve a full or partial outage? If hardware is likely damaged, how quickly can it be replaced? Etc. Suppose there is a fire in a server room and the servers are damaged. You have the LTO tapes as backups, but no server, and no LTO drive to restore it with. How many days will it take to get an LTO drive and from where? During this phase of planning a vendor and consultant list should be compiled.

 

3. Risk Rating

This can be included in the budgetary section, or done beforehand. Combine the risk assessment items and impact ratings and sort them. This is important. You should implement a recovery plan that encompasses as many items as possible. By sorting, it makes the budgetary step easier when you need to cut coverage because of costs. 

 

4. Methods

There are many methods to ensure data continuity in certain scenarios, but be careful as there is rarely a one-size-fits-all approach to backups.

Mirroring: Is designed to mitigate single-point hardware failures. If your database server fails, having a mirrored server will ensure your data is available. Mirroring at another location may also solve router, switch and connection issues for external clients. Mirroring does not protect against corruption, viruses, user error or malice. On-site mirroring does not protect against theft, fire, flood, etc.

Removable Backup Media: Backups to media protect against issues such as corruption, viruses, user error or malice. If you leave these items on-site they will not protect against theft, fire and flood. If they are taken off-site, it will take longer to retrieve your data in case of loss. With backup tapes, hard drives and cds, the backup data itself is typically a day or older. If this method of backing up is your only method, be sure your business can survive with older data. Be sure to have multiple days of backups, or a weekly backup with incremental backups per day. Often times users will not report data loss until days after the event, by which time relevant backups have already been overwritten with newer, useless backups.

Non-removable Backup Media: Items such as NAS (Network Attached Storage), DAS (Direct Attached Storage) or SAN (Storage Area Network) can be used to backup servers, virtual machines and data. The issue with these is that they are not removable. This will not protect against theft, fire, flood, etc.

Be careful of proprietary systems used to backup your data. Be sure to audit your recovery scenario regularly to ensure your backups can be recovered. Companies go out of business, and items are discontinued. Do you have any backups on jazz drives? How difficult would it be to recover if you had to find a new jazz drive? Don’t know what a jazz drive is? Exactly!

 

5. Budgetary Concerns

With your sorted list in hand, you can now plan for the items you need to mitigate any disasters. Protecting against many scenarios may prove to be prohibitive in cost. If you do not make the budgetary decisions, be sure that your list is as comprehensive as possible. It is up to IT to determine the impact of all scenarios, and it is up to the budgetary members to determine how much they want to spend. If they say no, you have at least outlined all the possibilities.

In your cost analysis, include replacement or redundancy items such as:

1. Backup Storage (Tapes & Tape Drives, Hard Drives, CDs)
2. Backup Servers, NAS, SAN, DAS
3. Mirrored Servers
4. Redundant Connections (Internet and Cabling)
5. Backup Routers, Switches, etc

Part of your impact analysis should include what is damaged or lost. If you have the tapes, but no tape drive, you will need to replace it in order to retrieve the data. Make sure you have the ability to read your backups when you need. If it takes 3 days to ship a new tape drive, but the cost is minimal, consider having a backup tape drive in stock.

While they are numbers out there for determining how much to spend on a backup and recovery system, you should make your decisions based on the impact and risk. If your data is your business’ main asset, you should spend a larger chunk of your budget to protect it. If time is critical in retrieving your data, the solution may include keeping extra hard drives, servers, router and switches in stock. If time is not an issue and an outage can be handled for days, you can order items at the time of recovery.

Determining budget can be a mix of preventative costs and the cost of downtime to the business (lost sales, lost productivity). Ideally disaster scenarios should have a cost to the business attached to them. If a server failure results in $0 productivity for the day, the overall impact can be many thousands of dollars per day – that fact alone may convince management to have a redundant server available.

 

6. Deployment and Testing

Don’t forget this step! Backups are useless unless they can be recovered. Take a weekend to simulate a likely recovery scenario. You may be surprised at all the “gotchas” when recovering data. Common stumbling blocks include not backing up database logs that are critical to recovery (ex. Exchange Server), or recovering to dissimilar hardware (Ex. RAID5 on a different controller).

No Comments |  Print This Post |

Last night Malwarebyte’s AntiMalware program detected a false positive of the Atapi.sys driver and associated registry keys. As you may know, Atapi.sys is required by the storage system in Windows, and as such deleting it will render the system unbootable.

If you have the reboot on error checked in your system properties, your system will continuously reboot itself without giving an error. If you have a stop on reboot option checked, you will see the STOP error 0x0000007B.

I have included a zip file with the following registry keys and atapi.sys (5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)) taken from a fresh install of Windows XP SP2. Apparently it only affects SP2 installations.

Registry Keys Infected:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\atapi
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\atapi
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\atapi

Files Infected:
C:\WINDOWS\system32\drivers\atapi.sys

A few ways to fix this:

Use another machine to load Atapi.Sys and Registry Keys

1. Install the hard drive into another Windows computer and copy the computer’s good Atapi.sys driver (Windows\System32\Drivers) to your hard drive. Or download the XP SP2 one here.
2. Put the hard drive back into the original computer and select “Last known good configuration” to boot – this will restore the registry keys.
3. If the “Last known good configuration” doesn’t work, you may try editing the registry hive of your installation via another computer. Put the hard drive into a second machine and load the hive within that machine’s registry editor. If you are uncomfortable doing this, create a LiveCD of Windows (below).

Make a BartPE or LiveCD of Windows

1. Go to http://www.ubcd4win.com/howto.htm and follow the instructions to make a LiveCD of Windows. You can also use BartPE, but the following instructions may be different (http://www.nu2.nu/pebuilder/).
2. From a known good machine, export the above 3 registry keys to a USB drive.
3. From a known good machine, copy the atapi.sys driver from Windows\System32\Drivers to a USB drive. Or download the XP SP2 keys and atapi.sys here (zip).
4. When the LiveCD loads (this will take a while), attach the USB key to the machine. Copy the atapi.sys to your machine’s Windows\System32\Drivers directory.
5. In the LiveCD’s Windows environment, go to: Start>Program Files>Registry Editors>Regedit (remote).
6. You will be prompted to select a user from your machine to edit. Most likely it is “Administrator.”
7. Go to File and Import in the registry editor.
8. Import each of the 3 .reg keys you exported from a known good machine.
9. Restart your computer, taking out the LiveCD.
10. Everything should work.

Use Windows Repair

I don’t like this option because it does not always work. You may also need to reinstall or fix some programs after this procedure.

1. Put the Windows XP disc into the machine.
2. When the machine boots into the Setup environment, it will give you the following options:
   To setup Windows XP now, press ENTER.
   To repair a Windows XP installation using Recovery Console, press R.
3. Press ENTER, not R.
4. On the next screen, it will detect a previous installation and ask if you want to repair it. Choose to do so.
5. Windows will go through the setup by reinstalling all default options and drivers. You will need your Windows XP key.

14 Comments |  Print This Post |

Most System Administrators use a hardware firewall to block IP addresses from accessing their network. Co-located servers do not always have the advantage of utilizing a hardware firewall. Software firewalls can often be expensive.

As you may already know, Windows 2003 lets administrators control IP access from the configuration panels in SMTP and IIS, among others. But what if you want to block an IP address from all services with only one motion? This is where the IP Security Policy Management snap-in comes in handy.

Configure the IP Security Policy to block your first IP address

1. Click “Start” and “Run” – type “MMC” and press OK.
2. In the MMC, click “File” and “Add/Remove Snap In.”
3. In the “Standalone” tab, click “Add.”
4. Select “IP Security Policy Management” and click “Add.”
5. Select “Local Computer” and click “Finish.”
6. Close the “Add standalone Snap-in” window and click “OK” on the “Add/Remove Snap-in” window.
7. Now that you are back in the MMC console, right-click on “IP Security Policies on Local Computer” in the left-hand pane and select “Create IP Security Policy.”
8. Click “Next.”
9. Enter a name (ex. IP Block List) and description into the boxes and click “Next.”
10. Leave “Activate the default response rule” checked. Click “Next.”
11. Leave “Active Directory default (Kerberos)” checked. Click “Next.”
12. Leave “Edit properties” checked. Click “Finish.”
13. The Properties box should be open.
14. To add your first IP address, click “Add.” Make sure “Use Add Wizard” is checked beside the button.
15. Click “Next” when the “Create IP Security Rule” wizard opens.
16. Leave “This rule does not specify a tunnel” checked. Click “Next.”
17. Select “All network connections” under Network Type (unless you want to specify by adapter). Click “Next.”
18. You are now at the “IP Filter List.” The “All ICMP Traffic” and “All IP Traffic” options will not meet our needs; we will need to add another. Click “Add.”
19. Name the IP Filter List (ex. Blocked IP List) and enter a description. Click “Add” to enter the first IP address to block.
20. The “IP Filter Wizard” will pop up. Click “Next.”
21. This will be the first IP address or IP range we enter to block. Enter a description (I usually enter the IP itself) and make sure “Mirrored” is selected below. This will ensure packets to/from are blocked, allowing you to create one rule instead of two. Click “Next.”
22. Keep “Source Address” as “My IP Address” and click “Next.”
23. Under “Destination Address” select “A specific IP Address” or “A specific IP Subnet.” If you select “Any IP address” it will block all IPs!
24. Enter in the IP address in the fields below and click “Next.”
25. Under “select protocol type” choose “Any” (means “All”) unless you specifically want to block from RDP (Remote Desktop), TCP or UDP, etc. Click “Next.”
26. Click “Finish.”
27. Now that you are back to the “IP Filter List” click “OK.”
28. You will be back in the “IP Filter List” list in the Security Rule Wizard – make sure you select your new “Blocked IP List” and not “All IP Traffic” or “All ICMP Traffic.” Click “Next.”
29. You will be taken to “Filter Action.” The lists: Permit, Request Security (Optional), and Require Security will not meet our needs. Click “Add.”
30. In the “IP Security Filter Action” wizard, click “Next.”
31. Select a name (ex. Block all Packets) and click “Next.”
32. Select “Block” for the filter action behavior. Click “Next.”
33. Click “Finish.”
34. You are back to the “Filter Action” list. Select your new list (Block All Packets) and click “Next.”
35. Click “Finish.”
36. You are back to your IP Security Policy list (Blocked IP List) Properties. Click “OK.”
37. Back in the “IP Security Policies on Local Computer” snap-in, you’ll need to assign the new policy. In the right-hand pane, right-click on your new list (IP Block List) and select “assign.”

To make it easier the next time you wish to block an IP address, save the MMC Snap-in configuration as a shortcut. Go to “File” and “Save As” and save it on your Desktop or Start Menu.

To Block Additional IP Addresses

1. Enter the IP Block List snap-in you saved.
2. In the right-hand pane double-click your IP Block List.
3. Under “IP Filter List” select the newly created “Blocked IP List” and click “Edit.” Make sure “Use Add Wizard” is checked.
4. Under “IP Filter Lists” select your “Blocked IP List” (not All ICMP or IP Traffic) and click “Edit.”
5. You are now in the “Add IP wizard” area. You will see the first IP address you blocked in a listing under “IP Filters.” Click “Add.”
6. Follow all previous steps to add the IP address you wish to block. Once finished, exit all dialog boxes.

You may need to restart the server for the settings to take effect.

1 Comment |  Print This Post |

This past week I’ve been busy battling 29 different IP addresses that have been attacking a server that I maintain.

In my effort to rid the world of this behaviour, I recorded the IP addresses, found out as much information as possible, and then blocked them.

Locations of the IP addresses:

• 12 – China
• 9 - United States
• 5 – Canada
• 1 – Netherlands
• 1 – Vietnam
• 1 – Japan

Compromised Operating System:

• 29 – Windows 2003

Compromised Web Server:

• 29 – IIS 6

Percentage without a Firewall:

• 100%

Twelve of the IP addresses were associated with specific companies running their own dedicated server for email, ftp or a website. I decided to call or email each company to let them know their server was compromised. Most were grateful that someone took the time to notify them. By the end of the week, 8 of these servers were considerably more secure! One of the companies I called was a Canadian computer store. The person I talked to had mentioned their server was slow and bandwidth usage was high for about a week.

These servers were compromised through poor security practices. Many did not have a firewall due to co-location requirements, and others did not have a firewall due to email and ftp not working properly when it was enabled. Clearly they did not know how to properly configure a firewall to let DNS, SMTP, POP3 and Passive FTP in/out.

I find one of the biggest problems with Windows is that it is too easy to set up and administer at a basic level. Because of its ease of use, the technical knowledge of the person setting it up doesn’t need to exceed that of a typical desktop user. They fail to take into consideration items such as security, assuming the operating system takes care of it.

1 Comment |  Print This Post |