Blog Host: Bob Konigsberg - Network Evaluation READ ENTIRE BIO

Every one of us has a collection of personal war stories, as well as those we've collected from colleagues, trade shows and Web sites. So I'd like to share some of mine which are network- and/or network security-related. I'll try to keep them short and to the point.

Director of Network Operations is very picky about who may and who may not have domain administrator status. Until his 14-year-old son comes to visit for the summer, and he makes him a domain admin.

As mentioned in yesterday's blog, I was working in the server room when a bunch of people came in demanding to know what I'd done. Nothing, in this case, but the entire set of racks behind me had their power strips daisy-chained, and the one more server added two days before it was finally too much for the circuit breaker to take any more.

Customer Service department complains that someone has run up over a $1000 worth of overseas phone calls, and they don't want to pay the bill, they want to find the culprit. Investigation determines that someone in Engineering learned about an outbound modem pool during a cooperative support case and had been using the modem pool to dial up to a BBS (yes, they still exist) in England.

Main Data Processing group complains that communication with the England-based data center slows down every day between 8:00 a.m. and 9:00 a.m. Investigation finds that an engineer visiting from the U.K. had been connecting back to his Unix workstation in London and using X-Windows to forward the display across the trans-Atlantic link, so that he could surf the Web from his office in California.

Server for one department has been trashed. The department is pretty sure that it's a disgruntled worker who has just been let go. Investigation showed (to my personal satisfaction) that it was indeed the person suspected. But none of the systems in the chain are using NTP/SNTP (Network Time Protocol/Simple NTP), and all of the time stamps disagree over a range of about two weeks, making prosecution impossible.

During a routine "war-dialing" sweep of all company phone numbers, a modem is found that allows direct pcAnywhere control (with no password) of a desktop computer in Toronto. When I connected to the system using pcAnywhere, it took several minutes to get the user of the machine to stop trying to kill the Notepad window I was opening to identify myself. I typed in my name, position and phone number (through the company switchboard - always remember your identification script) with extension. Turned out that it had been set up by a sales engineer in British Columbia to provide her with desktop support. But he had neglected to tell her that.

Shortly after joining a new outfit, I got a call from the on-duty operator reporting that our ISP had called to notify us that we were being used as a Smurf amplifier. A Smurf attack is performed by spoofing (forging) the source IP address of your victim and sending a ping (echo request) to a broadcast address (1.2.3.255) to get as many systems on that network to respond to the victim directly. So why were we vulnerable to this? Investigation showed that the VP of Engineering didn't like the response time from IT for firewall requests. So he demanded and was granted wide-open access inbound and outbound for his entire department subnet.

While examining firewall logs at a Midwestern company, I noticed a lot of P2P probes from a network a few thousand miles (and divisions) away. Investigation took several people in several different divisions because I didn't work with the remote division at all. But I could probe the remote machine, which showed that the only listening port on that system was for P2P file sharing. The system did not even respond to a ping. Investigation showed that the machine belonged to a director-level manager in a publishing company (who should have known better), who was running a massive music piracy operation from her office. I'm not sure, but I don't think she works there any more.

One of my painful lessons at one company was my attempt to suggest (not mandate, just suggest) by companywide e-mail, that everyone change their passwords from the default (first initial, first five characters of last name) to something a little better. The director of desktop support told me that this was not going to happen "because it will just tick off our users." (Lesson: If upper management doesn't care, you're wasting your time).

During an audit of an ISP, I found spam being sent to people that used their "internal" name as opposed to their "external" name. Tracking the e-mail headers back to their origin, I found that they led to a software development company. Bringing this to the attention of my customer (the ISP), they identified the last name of the sender as belonging to one of their customers. It finally turned out that the customer's son had broken into one of the ISPs systems from his father's computer, harvested the internal e-mail names and was sending his spam to advertise his new sideline of selling real estate.

One of my current investigations is beginning to look like some sort of back-channel that uses DNS ports. The IDS has caught these and flagged them as DNS violations. Problem is, looking at the packet data, it looks nothing like DNS, so I'm suspicious of it. But so far I've found nothing relevant on Google. Problem is, the source system is behind a proxied wireless gateway, so that's going to take more digging to find -- but it should be interesting, whatever it turns out to be.

Happy securing and networking, Folks...Bob
Posted by Bob Konigsberg Babes in Toyland
20 OCT 2005 00:00 EDT (04:00, GMT)
Disclaimer: These blogs are supposed to be about security and network management. This blog is not -- or not directly. It's more about the sorts of tools your support staff should have to make their lives easier and to make that staff more effective in their jobs. The signs on the sides of my truck say "Network Security" and "Troubleshooting" as the key bulleted items. The reason for this is that I've been involved in so many cases that started out looking like one and turned out to be the other. You never know until you start digging. So keep an open mind as to the possibilities.

Someone said, "The only difference between men and boys is the cost of their toys". That's not quite true, and, of course, not all of us in this business are men. I've known several female techies who loved their "techie-toys" just as much (or more) than the guys did.

So let's think (out loud) about favorite toys. I'm sure many of you will know about some that I've missed, but this should be fun anyway.

Network cable testers: These will check copper and fibers for distance, DB loss, miswirings, breakages/damage, etc. My pet Fluke one can come with several "ID" adapters for the other end, which help to identify some miswirings and help you tell one jack from another.

Adapter cable testers: These will check adapter cables that go from USB(A/B), BNC, Centronics, DB-25, DB-9, RJ-11/14/45, DB-15, AUI (also 15 pins, but larger connector/lower density) and FireWire in any combination you care to name. They will also tell you which pin on which connector is wired to which pin on the other connector.

Preface for the following two: The National Electrical Code specifies that any circuit that is under continuous load (three hours or more per day) for 15 and 20 amp branch circuits shall not exceed 80% of its rated capacity. This means that a 20 amp circuit cannot carry more than 16 amps for three hours or longer and 15 amp circuits cannot carry more than 12 amps for three hours or longer. For circuits larger than 20 amps (30+ for most purposes), the capacity of the breaker must be at least 125% of the rated full-time load. I've seen entire server rooms brought down because of daisy-chained power strips (which is a no-no from the fire marshal's point of view anyway) that finally overloaded the breaker. I've also seen data center management who, when informed that they were close to the breaking point load-wise, refused to schedule time to rearrange the loads because "we can't spare the time." I sometimes wonder if they ever got "bitten" by that.

Clamp-on ammeter: These will measure the AC amperage load of any device as long as it's clamped around ONE of the conductors. It's typically used at the breaker panel by an electrician, but I've made up some pigtails where each of the conductors has a loop sticking out and a male and female connector on the ends so that I can insert it into the power cord and measure the load of a single device or group of devices. Here's a mini-picture:

=|----8----|=|

Digital load read-out power strips: These are usually bolted on to each rack of equipment and provide continuous digital readout of the current load on the power strip.

Circuit breaker locators: These handy little guys are two-part. One part plugs into a wall socket (110v only) and generates a tone on the power line (assuming that the circuit is live). You take the other end to the breaker panel and scan it over the breakers listening for the tone. If you don't find the tone, turn up the sensitivity until you do hear it. If you get the tone over several breakers, then lower the sensitivity until it's just on one breaker. If you still don't find the tone, or it shows on all breakers, find another breaker panel.

"Toner and banana": This is originally a telephone installer's tool, but it works for copper network cabling as well. Similar to the breaker locator above, it's two parts: a battery-powered tone generator that has both alligator clips and an RJ-11 plug to connect to one end of the phone line, and a battery-powered pickup that you run along the line of jacks and cable until you hear the tone. Saves a LOT of time.

Mini-Mag-Lite: 'Nuff said.

Quickie labeling machines: You know the ones. They range from 1/4" to 1" tape width with a variety of tape and print colors. Some also have USB ports so that you can drive the printing from your PC.

Tweezers: For straightening out bent pins in connectors. If you're anywhere near my age, you'll also need a….

Stand-up magnifying glass: I found a neat one that folds as a square and unfolds to stand up on its own. I've seen them in Fry's Electronics, Radio Shack, various mail-order magazines.

Universal RJ crimping tool: I've seen (no joke) the price range for these from less than $20 (Radio Shack) to over $400. Decent ones range from roughly $40 to $250. Do not buy the Radio Shack ones, as they do not enforce the proper crimping pressure on the pins. I also always use my cable tester on all crimps made at the time to eliminate the possibility that I've just introduced a problem into the environment.

Cable ties and adhesive pads: You know what they are. Question is: Do you carry some with you? You can also get cable ties with little label areas to identify a cable with your quickie label machine.

Roll of "hook and loop" ribbon: Used like cable ties, but you can undo and redo them later.

Staple gun for 8 mm round cable: Helps make visible wiring neater.

Screwdrivers: You should have at least #0, #1 and #2 Phillips and 3/16" and 1/4" or better flat blade ones. I prefer Klein because of the better quality. (They are usually sold at Electrical Wholesale houses, but I've also seen them at Home Depot.)

Pliers, wrenches, wire cutters, torx drivers, punch tools (more for telephone work)

Spare straight-through and crossover cables: Cat 5e or Cat 6 (or whatever is next).

Serial console cables: Most people are familiar with the Cisco light blue ones. But there are many brands of equipment that don't meet that standard -- including Cisco equipment from acquisitions (Arrowpoint comes to mind). I've set up my own standard based on the old US Robotics serial cable -- simply because I worked at 3Com at the time and the USR setup used all possible pins within an RJ-45, whereas the native 3Com ones did not. By setting it up this way, I've got a bunch of different style connectors (DB-9M, DB-9F, DB-25M, DB25F, RJ-45) that can be plugged in almost any serial port and to your PC (desktop or laptop) and gain instant connectivity to the console port.

10/100 Mb Hub: Not a switch, a hub. For use in sniffing traffic at odd places where you don't have access to a SPAN port. Netgear still sells these as a 4-port or 8-port unit for roughly $20-30.

USB-powered floppy diskette drive: Fewer systems these days are made with a floppy drive installed. These can still help you out of a jam. You can also still use WinZip to make a multi-volume floppy-diskette-based zip file that will span as many of these as you want.

USB external hard drive: For making copies (Ghost and such) for configurations, documents and forensic images (requires special boot CD).

Wrist grounding strap: If you're working on electronic equipment and you've got to take out parts, ground yourself to the chassis.

Generic multi-voltage power supply: Test replacement for equipment that uses those "calculator-type" power supplies. They typically have a switch on top to control the output voltage and polarity, and then a bunch of different style connectors for many devices. Just note that if you measure the output voltage not under load, it will be higher than you expect. Don't worry.

Small vacuum: Heat buildup is the enemy of all electronics. Computers, switches, routers, television sets, stereos -- doesn't matter. The buildup of dust acts as an insulating blanket that will accelerate the aging of your equipment, and in some cases cause strange failures that are difficult to diagnose. In my shop, I actually use filtered, dehumidified compressed air, and I've got a tank that I can take with me. The problem with that is if dust is a serious problem, you can't be blowing it all over, so the vacuum approach will still help -- and be neater. I just found a 1-gallon shop vacuum at a local Ace hardware store for about $40.

High-quality surge and artifact power filter: Most people think of surge protectors as keeping lightening strikes from burning up their PC. But electrical artifacts can come in the ranges of seconds, 10ths and hundredths of seconds, milliseconds, microseconds and nanoseconds. Each of those different duration artifacts can have different effects on your equipment depending on the length, spacing, repetition and frequency of the artifact, and the quality of the power supply on the equipment. The best ones are made by APC and Tripp-Lite: It's amazing how many people (and networks) suffer problems because of dirty electrical power. Since this is part of your toolkit, it's just to plug the system under suspicion into to see if cleaning up the power solves the problem. I've been called in for a few cases where the customer was sure that someone was deliberately crashing their networks, and the culprit was dirty electrical power. The solution is not one of these power strips, it's a UPS. The power strip simply helps you identify when dirty electrical power is the problem.

Just be sure to buy the very best quality tools you can afford.
Posted by Bob Konigsberg Dirty laundry
19 OCT 2005 13:48 EDT (17:48, GMT)
Let's discuss how to make yourself unpopular with your co-workers. This is not the sort of advice you read about in Miss Manners or Live For Success. Today's subject is internal auditing.

Putting up a well-configured firewall is a great idea. It is good authentication for the VPN and (if you still have it) dial-up pool. But the firewall and other things work even better when there are fewer things inside to compromise. It also works better when nobody has circumvented your protections by some other mechanism. So we'll cover a lot of these things. Let's start with the firewall. I'm performing a two-step process for one of my current customers and have done this in the past for others as well. Intrusion and extrusion testing. My laptop is sitting inside the firewall, with no special permissions of any sort -- just a normal workstation from the firewall's point of view. Using another system, I make an SSH connection to my Linux box at home and proceed to perform a full 64K TCP Nmap port scan of my laptop -- with a sniffer running on the laptop. That's the intrusion test. A simple scan of one system with no special permissions -- just to see what (if anything) does get past the firewall. Now, I reverse that. Using a port scanner on my laptop, I scan my home IP address, knowing that my Snort system will pick up anything and log it for later analysis. So right off the bat, I've got a picture of what the effective permissions "mask" looks like for inbound and outbound access. We'll assume that this will be dealt with later. If you've got an organization with multiple geographic sites, you need to check for rogue Internet connections. There are several ways of doing this. Here are a few:

• Call up the remote site and ask for whoever is most technical at that location (the Alpha Geek) and ask him or her how they get to the Internet. Nothing like the simple approach.

• Log into a remote router or other manageable device to which you have access and do a traceroute to a public site.

• Log into a remote router or other manageable device, ping your external firewall, and look at the logs to see where it came from.

• Set up a permit-only ACL (Access Control List) on their router that permits, in turn: Internal Organization traffic IP range(s), then wide ranges belonging to popular Internet sites (MSN, Yahoo, Google, ESPN, Ebay, AOL), then an "everything else" entry. Wait for anywhere from a few days to weeks and see what hits you get against the access list. If all of the popular and "everything else" entries are zero, you've probably got a rogue Internet connection -- assuming that it's not documented and official. You did look first, didn't you?

• Check for illicit modems. There are still plenty of them out there. You'll need what is called a topology report (in the U.S., anyway) from the phone company(ies) lists all of the incoming phone numbers/lines that you are paying for. If you talk to whoever handles your organization's phone bills, they may already have a copy. There are some very fancy dialing checkers out there, but I still use ToneLoc (by MuchoMaas and MinorThreat), which is fairly ancient, but works just fine for me. It's got checks in there for various PBX signals/tones, but you don't need to look for them in this effort. It will dial in sequential or pseudo-random order any block of phone numbers you give it. It's purely a DOS (No, not Denial Of Service, Disk Operating System) application -- so you'll have to run it in a command line window. Interesting sidebar though: I once tried to disassemble the code for ToneLoc, only to discover that the authors have it wrapped up in its own encryption scheme. Very clever and discouraging enough to make me not bother. I'm sure someone reading this will think "lamer" to him or herself.

• Check for rogue wireless access points. Get a laptop equipped with NetStumbler, make sure NetStumbler works with whatever wireless card you have in it against a known wireless access point -- even if you've got to make a trip to Starbucks or similar public place to test it. Point of clarity here: NetStumbler is not a wireless security tool; it's more of a wireless survey tool. There are ways to configure a WAP to evade detection, but if the owner has done a good job of it, then it's not that great a threat to your security -- relatively speaking anyway.

• Check for cellular modems? Good idea, but I have no way that I'm aware of to do that. If someone in the reading audience does know, please email me here. Just because I don't know how to deal with such a thing doesn't mean that I don't regard it as a potential threat.

• Check for outbound-oriented remote desktop apps. GoToMyPc and Mionet (mentioned in a previous blog) make an outbound connection to a remote site, which effectively allows inbound control to a system inside. GoToMyPc directs all traffic to "poll.gotomypc.com." I'm still investigating Mionet, so I don't have all the data yet.

• Check your firewall for inbound remote control connections as well --mentioned in the same blog as just above.

• Do a port scan internally for the following ports (there are more, but you've got to start somewhere):
  • SSH (22/tcp)
  • Telnet (23/tcp)
  • RDP (3389/tcp) (Microsoft Remote Desktop Protocol or Terminal Services)
  • pcAnywhere (5631/5632/tcp/udp)
  • Dameware (6129)
  • Citrix (1494)
  • Laplink (1547)
  • CarbonCopy (1680)
  • BackOrifice (31337, 8787, 31338, 54320, 54321, )
  • SubSeven (1234, 1243, 1999, 2773, 2774, 5873, 6667 (IRC), 6711-6713, etc.) (You can get free scanners for the bad guy ones)
  • VNC (5800/5900/tcp)

If you want to look up more, go Treachery.net, which uses the Neohapsis port listings. If you've got the client side apps, check to see if any of them let you in with no credentials or generic domain credentials.

Check for open file/folder shares. I still use the Legion tool, but it's not complete, so I've supplemented it with some Perl scripts and Batch files. The last few years, I've bought (and use) NetScanTools and NetScanToolsPro -- both from NetscanTools.com, which also scans for file shares and the like. One Midwestern company I audited a few years back had over 6 million writeable files discovered -- mostly from people sharing the root directory. But of those 6 million, between 400-500 of the files had names suggestive of confidential information. The criteria for these were as follows: 1st pass *.doc, *.xls, *.ppt, *.wpd (Word Perfect), *.vsd (Visio document). This reduced the list considerably down to the 10s of thousands instead of millions. The second pass was to check the file name list for words like "budget," "evaluation," "project," "review," "termination," "schedule," "offer" and others. This was what netted us the 400-500 filename list. I recall someone asking me at the time what those documents contained. My response was "Are you kidding?" Confidentiality aside, there simply isn't time to look at that sort of thing, unless you're actually looking for evidence of criminal wrongdoing -- in which case that's the wrong approach anyway.

Perform a port 80 (and its variants 81, 82, 83, 85, 88, 8000, 8001, 8080, etc.) scan to look for rogue Web servers. Many of them will be printers, print servers and authentication-required devices of various sorts. If you get prompted for a username and password, and simple efforts don't get you in, that's good.

However -- just to trash the previous paragraph -- there is a Web site that provided the manufacturer's default account settings for many, many different types of equipment. Phenoelit.de is the place to learn how many people in your organization have not bothered changing the manufacturer's default username/password. Pretty scary overall.

There are lots more things you can do to audit the inside of your network. But if you go through the list above, and add the other things that occur to you as you learn what is really going on in your network, you'll have made a really good start! And if you follow through, I applaud you.
Posted by Bob Konigsberg Nobody knows the trouble I've seen
18 OCT 2005 20:17 EDT (00:17, GMT)
The business of troubleshooting can be tricky. There are tips, hints, guidebooks and articles about various aspects of troubleshooting by the score (as in four score and seven years ago...,but never mind, that was Abe's blog), but they all seem to miss a key point.

That is: "What does the customer mean by their complaint?" This is the part that technical folks often despair over because the vocabulary that the end user uses might sound the same as the technical folks use, but the chances are great (bordering on certain at times) that the meaning is completely different.

Unless you are dealing with colleagues whose technical background you are familiar with and can automatically adjust for, dealing with end users (customers -- whether or not any money changes hands directly) can be very frustrating for many technical folk who have not taken the time to learn how to properly handle them.

When a customer starts out explaining a problem, let him or her finish. Don't interrupt. It's frustrating to be treated like a kindergartener. If you are unfortunate enough to get a rambler (who starts out on a computer problem, and somehow moves to why his sister now lives in Poughkeepsie), then you will have to interrupt carefully. "Excuse me sir/ma'am" until you have their attention, and then ask them to get back on track with respect to the problem. As long as they are sticking to the problem, though, let them finish. You may be lucky enough to have them identify the actual problem as they struggle to explain it. Think of it as being a guest in someone else's house -- they get to make the definitions and rules. And it's good public relations. As sick as we techies are of dealing with the clueless, they are just as sick or more so, of being treated like blithering idiots.

In the process of listening, you may discover what you suspect the real problem to be. Don't jump in, let them finish. Their continuing on gives you two things:

1. You can better gauge their level of technical knowledge to assess the accuracy of what they are telling you, and
2. You can think of the best way to ask the questions that may elicit better information from them -- in terms they can understand and relate to.

Get used to asking extentional questions in their area of competence -- their personal universe.

• When did the problem start?
• Was there ever a time when it didn't do this?
• Does it always do this? (Clouds of smoke and the fire engines aside.)
• Were there any events you can think of that might be related to the problem?
• Have you or anyone else tried to fix this problem? (I can hear you groaning now.)
• If so, what did you or they do?
• If you don't know what they did, would you give me their contact information?
• Is there any other odd behavior that might be related to this?
• Does anyone else have this problem?

One of the key factors in solving any problem is getting the right information to set you on the right track toward solving it. Truth be told, I'm a certifiable tool fanatic. It doesn't matter if it's wrenches, screwdrivers, woodworking tools (my hobby), test equipment, software tools, etc., if there is something made to solve a particular problem more easily, I'll buy it or build it if it helps me (unless the cure costs more than the disease). In this case, we'll stick to tools that provide information relevant to solving computer and network problems. Some of these tools are free, some are shareware, some cost money, some are built into the operating system or part of the overall environment. What they have in common is that they provide some sort of information to help you narrow your search for the solution to the problem you are facing. If you choose not to use them because you're sure that they would not be helpful, then you're being just as ignorant (and arrogant) as some of your customers.

So here is a simple list -- there are multiple tools available for many of these, but I'll just list one or two.

• Event or system logs -- comes with the respective operating systems
• Pop-up messages -- teach your customers to write down the entire message before they dismiss it
• Process information -- Task Manager on Windows, ps on Unix/Linux systems, ProcessExplorer (SysInternals.com)
• Network listening status -- Vision (Foundstone/McAfee), TcpView (SysInternals.com)
• Network traffic -- Network General Sniffer, Ethereal, WinDump/TcpDump
• Registry activity -- Regmon (SysInternals.com)
• File system activity - Filemon (SysInternals.com)
• Disk drive status (error counters, temperature, etc.) -- DriveLED (OO Software)
• Chkdsk/Scandisk(Windows) / fsck (Unix/Linux)
• Startup info (Windows) -- HijackThis (Safer-Networking), Auto-Runs (SysInternals.com)
• Antispyware scanner -- Spybot Search & Destroy, Ad-Aware, Pest Patrol, CounterSpy, Microsoft Anti-Spyware, WinPatrol, etc.
• Logs from possibly related systems and services -- name servers, firewalls, intrusion detection systems, routers, switches
• Cable tester(s) -- Intermittent cables have caused me a fair number of problems. I happen to like the Fluke product line, but there are many other brands out there that also do a fine job.

To start with, get a copy of whatever documentation does exist on the network and where/what the organizations are/were and how they fit into a picture of the network. Then, get a copy of the latest org chart (some places, these are regarded as confidential), the company phone book, a complete list of routing tables from the local network and the last two years' worth of press releases.

Go through the contact list one by one, call up each person, introduce yourself, provide your contact information, explain why you are calling and request that, for security's sake, they call you back through an identifiable organization phone number. This way, you are not only learning people, you're enforcing good security practices and leading by example. This process will be called the "identification script." If you get a disconnect recording or some such, make note of it and move on. If you get the wrong person, then, after properly identifying yourself, explain what you're trying to do and whether or not they can direct you to the proper person. Take notes, and move toward any clues provided.

Go over the routing table entries with someone knowledgeable at your end. If you've also got a Visio diagram, so much the better. Identify all known end points, and correlate them with the initial pass of the contact information.

Now we get to the fun part. For all networks for which you don't have contact info, get an SNMP query utility (I like and use SolarWinds.net's suite of tools, but there are many others out there) and query all IP addresses with the "PUBLIC" and "PRIVATE" SNMP strings and wait for results. These results will contain a large number of printers and the occasional Windows, Unix or Linux machine. Create a full-page document that says "IF YOU FIND THIS PRINTOUT, PLEASE CALL SoAndSo at (111) 222-3456 (Printer IP Address)" in the largest font that will fit on one page. Install appropriate printer drivers as necessary, and configure at least one printer for each remote office; send the printout there. Sometimes you'll get a call back (sometimes irate) within minutes, sometimes not for days and occasionally never. Once you get hold of someone, ask them who does their IT support, and go through the rest of your identification script.

If your organization sells goods or services, see if the main Web site lists all sales offices. If so, call up each of them, go through the identification script and find out where their IT support comes from. Then call those people and do the same thing again (assuming you haven't already come across them already).

Now, start going through all of the press releases looking for signs of acquisitions, shutdowns, moves, divestitures, etc. If any of them appear on your lists, great! If not, call them up (or better yet, the organization's spokesperson) and go through your identification script.

Some folks are going to get upset at my tactics, saying that it looks too much like social engineering (which it is, to some extent), but I maintain that it's perfectly OK, as long as you are careful to stick to your identification script, allow people to get back to you after checking you out (some will, some won't), as opposed to pressuring for an immediate result (a common characteristic of social engineering).

I'm not bothering with the details of record keeping because every situation is different. What's important here is that each of you have more resources than you might think you have. In areas where the organization is physically close enough, I've done office and cubicle wandering as well. You'd be amazed at some of the things you learn that way, too! The key message is to ask lots of people lots of questions (don't forget the identification script), and even when direct answers don't provide what you're looking for, a lot of pieces (just like a jigsaw puzzle) will start falling into place and form a more complete picture.
Posted by Bob Konigsberg Put another log on the fire
14 OCT 2005 23:17 EDT (03:17, GMT)
Some years ago, I presented a paper at TISC (The Internet Security Conference) titled How and Why to Read Firewall Logs. That was about (or past) the end of an era when it was humanly possible to read one's firewall logs -- and most of mine were condensed for automated reporting simply because of time and space constraints.

However, it remains a subject near and dear to my heart (professionally speaking, that is) as a means of keeping tabs on what sorts of things are going on on one's network. To that earlier list, of course, one must add in IDS/IPS logs, VPN authentication (or failure thereof). As I've mentioned with other topics, most of you have resources that you don't know you have.

One of my favorite toys of late is the Cisco MARS product (formerly "Protego" before Cisco acquired them). This box will take logs from many different sources and correlate them into "incidents" which can be examined and drilled down into. It catches lots of stuff and, like an IDS, will have to be tuned to cut down on false positives. Network Intelligence and others make roughly comparable products also.

But what if you don't have one?

One of my favorite things to do, which I spend 15-30 minutes per day on (sometimes more), is to export different aspects of firewall logs into Excel and sort the data different ways to see if any patterns jump out at me. Recently, I just chose all outbound connections aimed at ports 1025 and higher. Sorting by destination port showed one group of heavy traffic with a destination port of 6667, which is IRC (Internet Relay Chat). Experience has taught me that most systems talking to an IRC server are doing so without the owner/user being aware of it -- that is, they're infected by a worm, virus, Trojan or bot, which is usually "phoning home" for instructions or to provide reports of some sort. When I found this set of logs, I ran mIRC (and IRC Chat client) to try and connect to each of the servers. There were three that refused my connection outright. So the systems connecting to them have already been reported to the desktop support group for immediate investigation. The clients going to IRC servers that permitted me to connect will also get investigated, but are slightly lower priority.

Peer-to-peer traffic is a problem as well. I've mentioned that in previous blogs. By sorting source and destination addresses and looking for known P2P port numbers, I've learned that any time a given system contacts a P2P server, it attempts to open a wide variety of ports. In return, the system contacted immediately tries to open many corresponding ports back on the originating system. One experiment I tried with BitComet showed that it momentarily opened dozens and dozens of listening ports when it was first brought up and then stopped most of them. As a result, I've gained more insight into some of their behavior.

Setting the firewall log monitor (Checkpoint, in this instance) to filter on popular attack vectors (NetBIOS, A/D, SNMP, HTTP/S, etc.) often reveals attack patterns, virus infected systems, misconfigured systems and the like.

Looking for broadcast addresses (particularly using a wider broadcast mask than is used internally) shows both attackers and systems with misconfigured subnet masks.
Posted by Bob Konigsberg Another brick in the (fire)wall (AKA Top 10 things to think about when running a firewall)
13 OCT 2005 22:18 EDT (02:18, GMT)
Well, it's almost firewall review time, so it seemed like a good idea to inflict this stuff on everyone else, so that you can share in the joy (or misery -- take your choice) of the process.

Firewall policies tend to be as varied as the organizations that implement them, but there are several general guidelines that make the process a little more structured and help prevent "rule rot" from setting in too deeply. Most of you probably immediately understood what rule rot was even if you've never heard the term before. For the rest of you though: Rule rot is what happens when new rules, exceptions, objects, etc., are applied to a firewall without first looking at what is already in place to maintain a consistent structure. At some point someone realizes that it's become an unmanageable nightmare.

I'm making the assumption of a GUI interface such as Checkpoint, Juniper, iPolicy and such. If you are using a Pix, router ACL, or other text-/console-driven management method, then you'll have to add comments to the configuration commands in a master document, and keep everything up to date in the master document no matter what!.

Consider this to be the Top 10 things to think about when running a firewall. It's not complete or perfect. But it will save you a lot of time and trouble if kept up. If rule rot has already crept in, then start at step 10 and work backwards.

1. Who wants the rule? What manager and techie are responsible for it? In various places, I've implemented a policy that anyone who wants an exception or new rule needs to provide written approval (e-mail is fine) from a manager at director level or higher. This doesn't represent too great an obstacle, but just enough to keep frivolous requests (and you WILL get frivolous requests) to a minimum, while not interfering with business.

2. For each rule and accompanying objects (name/IP address, etc.): Get and maintain the management and technical name and contact. If possible, put those names in a comment field with the object in the actual configuration itself. If the rule is unique to an individual group, then put the contact information in the rule comment itself.

3. For temporary or test rules (I've seen "temporary" rules that have been in place for years), put an expiration date in the comment field in addition to the contact.

4. Group like services together. All inbound Web servers should be in a single group with the relevant ports (usually 80 and/or 443).

5. Discourage non-standard ports without good reason. "I want to make it less likely for hackers and bots to find my Web page" is not a good reason. We tell those people that we will be happy to perform a Nessus vulnerability scan on their server to see if it meets acceptable minimums. Alternately, we suggest that they implement user IDs and passwords.

6. Discourage "I need all (65,000-odd) ports open because I have no idea what's really required" type of requests. For these, you can create a temporary rule (see #3), and then track the logs with the services in use to see what is actually needed. Then go back and tighten it down.

7. For legitimate non-standard port usage, consider grouping the oddball systems together under a single rule. In one sense, this violates the principle of allowing no access that is not absolutely necessary. But, if you check all relevant systems first (port scan them) and only one responds on TCP/7000 (and the others don't) and only two others (properly) respond on port 24555, then you may able to group them into an "oddball" rule safely. Just make sure that you recheck all of them every time you add a new non-standard port.

8. Many people will want special inbound access to their desktops. This sort of request can take the form of Telnet, SSH, Remote Desktop/Terminal Server, PCAnywhere, GoToMyPC, Mionet, VNC, LapLink, various databases and probably a whole host of other things. The answer is: "No. Implement a VPN." By the way, for those of you not familiar with it, Mionet is a relative newcomer to the scene. It not only allows remote desktop access, it also allows the user to "share resources" with those of his or her choosing. If this scenario doesn't scare you, you haven't spent enough time in security. By the way, technically speaking, GoToMyPc and Mionet are outbound applications, but the direction of access is inward.

9. Many enterprises simply allow all outbound connections without further restriction. There is very little reason to do this, and lots of reasons not to. First off, the overwhelming majority of traffic these days is HTTP and its partner HTTPS on well-known ports. If policy permits, you can allow SSH, FTP (active and passive), POP3 (dubious, as it can bypass anti-virus protections), SMTP (what's wrong with the regular mail server?), various forms of IM (Instant Messaging, in the forms of AIM, MSN, Yahoo, et al). Peer-to-peer (P2P) file sharing? Not on a bet. The college I help out gets LOTS of e-e-mail from the RIAA (Recording Industry Association of America), DMCA (Digital Millennium Copyright Association) and other copyright-protective trade groups who will provide a polite warning before serving a summons. You don't want your employer to have a "CNN moment." They are not much fun.

10. OK. Having inflicted all of this stuff on you, it's now expected that you will keep it up. Pick your interval (weekly, monthly, quarterly, semi-annually or annually), put it in your (common?) calendar application and review everything. Specifically the following:
    • For every object defined, validate that the contact names, numbers and e-mail addresses are still valid. Don't just look at them, call each and every person to make sure they are still part of your organization, haven't changed roles (promotions, moves, etc.) and still have responsibility for the object in question.

    • At the same time, verify and validate that the object and/or services to it still exist and are still needed. You might be surprised how many firewall holes I've found for servers that died years ago. Or for departments that got reorganized out of existence.

    • For all temporary/test rules, find out if they are still needed.

    • Ask the owners of these systems when the last time they were updated for OS, application, antivirus, etc. Ask for a specific date.

    • Consider scheduled Nessus scans, but that's sort of out of the scope of this blog.

    • Make all comments reflect the new data. I love to keep copies of configurations on my system, or a central server, but if we all get hit by a bus, the only copy of the configuration visible to the rest of the IT organization is the one sitting on the firewall itself. And that is where someone new is going to have to start.

This is for a customer who is a college, so a lot of the "open academic environment" mentality still applies. While the firewall has been locked down a fair amount, the key difference here is that every department, teacher, etc. can make their systems available via a Web page, database, SSH login, telnet (on campus only) and some oddball ports -- as if the normal range isn't enough. There are also courses thata have a lot of online content. This is an overall trend. My kids' high school distributes a lot of their homework etc. the same way. I've started the planning for a full audit of all exposed systems, but that's another story.

One of the bigger successes that has come about as a result of examination -- the IDP reports was a discovery of just how much nosy traffic was being allowed past the firewall. Case in point was MS SQL Server traffic. After tracking down the professors running those servers, it turned out that they were intended for on-campus students attending their classes only -- no exceptions.

Since most of the port scans were coming from the Asia-Pacific region (APNIC) and Europe (RIPE), I was able to set the border router to block large chunks of IP address space (like blocks of seven class A networks) from at least port 1433 by doing ARIN lookups to determine any blocks not on the North American continent. Additionally, any scans coming from China and Korea would get a class-C-sized range blocked from all IP access. I'd love to say that this solution was perfect, but it wasn't. It turned out that one instructor-run Web site makes a link back to somewhere in China, and I broke his page -- so I had to remove that particular ACL entry. Since then, I've also been adding other geographically identifiable network ranges that are highly unlikely to belong to on-campus students. The large nationwide ones (Comcast, AOL, for example) I don't dare block.

However, between blocking SQL and narrower ranges for everything, the IDS reports that I'm seeing are now producing results that show where the on-campus "threats" are coming from. The area I'm currently working on refining is the SYN Flood and distributed network scan reports. Since some networks are proxied out through a gateway, the IDP has no way of knowing that a given single address is actually an entire network, so it sees normal innocent traffic as a distributed attack. The trick now is to refine the filter definitions so as to not raise as many false alarms, while still being sensitive to an actual incident (which may or may not be an actual attack).

There is lots more progress to be made and, to quote Silicon Graphics, "serious fun."
Posted by Bob Konigsberg Utter CERTainty
11 OCT 2005 16:10 EDT (20:10, GMT)
Does your organization have a CERT (Computer Emergency Response Team) or anything else similar?

Have you ever discovered a problem, incident or other issue that isn't in your back yard? The larger or more geographically spread out the enterprise, the more likely this is to happen. Unless it is an ongoing clear and immediate danger, the first step is usually to contact a team member or someone in the location or organization in question, usually by telephone, IM or e-mail. If you can't make contact with that office within the first day or two, the perceived urgency tends to drop, usually pushed aside by something newer. In almost every organization I've worked in (either as an employee or outside consultant), the emergency contact list -- if one exists -- is hopelessly out of date. Why? Because no one "owns" it, and more to the point: No one maintains it. In almost every one of these places, I've found it necessary to rebuild the contact lists for names, phone numbers, e-mail addresses, network charts/maps. And where the list does have correct information, the individual people on the list often don't know each other, which can lead to turf wars when an incident does occur.

If you are in the security role/function/team, and you don't have a CERT (Computer Emergency Response Team) by that name or any other, form one. And make sure you get management support. Not just your management, but written (e-mail) permission of the manager of each person whom you want to join the team. An effective approach I've found is to "nominate" the candidate of your choice, but leave the final decision to their manager. You won't always get who you want, but by getting the manager's buy-in, in writing, the occasional emergency that sucks up more time than anticipated will continue to get support.

Let's now assume that you have management support. Find out who the IT people are in every location. Call them up and talk to them. Introduce and identify yourself, since in many cases, they may not know you from Adam (or Eve, as the case may be). Ask them who "owns" what and who should be contacted for what. Find out who their management is. Provide them with your contact information. In some cases, the organization has been so fractured that I've had to call sales offices to find out where they get their IT support from. In other words, the organization chart (and sometimes the IT/IS org chart) failed to provide enough structure to provide a clue as to where they existed. (In some cases, I've discovered small IT organizations that existed basically in a vacuum.) In any event, the next step is to put the entire list together and share it with everyone. In many cases, the contact may not be an IS/IT person, but simply a useful contact. Keep an open mind.

Now comes the maintenance. Keep them all in touch. Send out updates, notify them about new tools -- anything to keep the lines of communication open between you and all of them. Monthly conference calls and discussion groups build the idea of a team. At least once a year (preferably more) call up each person for some informal reason. Find out if there have been or are going to be organizational changes. Did they get a new manager? Then get the new manager's buy-in. Have they been promoted? They may or may not have time to devote to the team. Have any e-mail addresses or phone numbers changed because of technology changes (brand of mail server, VoIP, move to new facilities). Unless you have a HUGE organization, this maintenance should take less than an hour a month.
Posted by Bob Konigsberg IS/IT manners
10 OCT 2005 17:02 EDT (21:02, GMT)
Although my nominal topic is security, there's something more fundamentally important that is commonly seen in most people's workplaces, and on the TechTarget forums: R-E-S-P-E-C-T. A lot of people spend a moderate amount of time complaining about the inadequacies and incompetence of their colleagues. What I'd like to suggest (which won't cure the problem, but may cure some symptoms) is that when someone does something that seems hair-brained to you, ask them nicely (as if asking to be educated) "Geem Joe/Sophie/whoever, I'm a little puzzled as to why you did "'uch-and-such.' Would you mind telling me why?"

Now I can just hear the groaning..."What, me? Ask THAT idiot?" My key point here is that by treating your colleagues with respect, they will start to respect you. NONE of this takes place overnight, but if you bide your time and continue the practice, even the most difficult of your co-workers will start to respond, and start to pay attention to what you are saying. And who knows? You may learn something in the process.
Posted by Bob Konigsberg