Files, Backups & Junk

My backup tape ran out of room today. I wasn't terribly surprised. I've been trying to get management to notice the fact that files are growing at an astronomical rate & yet they still won't let me clean out any of the files from 10 years ago.

How do I get a handle on this problem? I've thought of a number of ideas & discarded most. I can't put limits on the amount of files the users keep or the size. Just reminding them they are over the limit on their email account is hassle enough. The Suits don't care about the size. They want everything to CYA & 'just in case'. I don't blame them - so do I.

I plan to pull all the files older than a certain date - trouble is, I've been asking for several years & still don't know what that date is. Three years sounds good to me. I'll create an 'Old' directory, that is read-only with the same file structure & put all the files I move off there. Then they can get to them quickly, if needed. Back them up a couple of times & they should always be available.

Clearing garbage won't help much or for long, though. Most of the files from more than a few years ago add up to about a third of the space taken up. Our old Word Perfect 6 files, although there are thousands of them, add up to only 3gb of space out of 150gb we're now using.

We have this catch-all directory where everyone has their personal files, files they share with others & all the group directories & files. Seven years ago, it was under 10gb. Since then we haven't done any serious cleaning, changing of the file structure or put any policies into effect to halt rampant growth. It isn't unusual for a single PowerPoint to take up 300mb. Everyone is scanning documents in now - we have workgroup scanners - & they're putting them in the same place as their regular Word documents.

Another option is a bigger backup. I'll have that ability in a few months, but again it is a short-term solution. Backing up that much data runs into the time crunch. There are NAS solutions, but getting the money for a larger backup was dicey.

I know this isn't an original problem & I've been looking at a lot of answers. Most of them are painful or expensive. All require that the Suits understand the issue - the hardest part - getting the executive buy-in.

Symantec Firewall & VPN

We looked over a bunch of different firewall & VPN solutions during the fall of 2004. We settled on the Symantec 5420 & 4420 because they seemed to fit our needs & our budget well. We could the 5420 firewall in place & then install a 4420 SSL VPN early in the following fiscal year. Symantec promised that in the third quarter of 2005, we could put the two devices together & have redundancy. A big cost savings would be only having to license users concurrently through each device once - not on both devices - for the service they used. I needed 100 firewall licenses outbound & 150 VPN licenses inbound.

We started by installing the 5420 Firewall (FW). An added bonus was FINALLY changing our Internal IP addressing scheme from a routable range to a proper, non-routable range, something I'd wanted to do from the day I walked in the door. (The horrors we inherit!) The project went well. The firewall wasn't particularly straightforward, but it wasn't terrible. At least it worked fine, although I had to buy 50 more licenses since there is no refresh built in. Any IP address sending out through the firewall uses a license. If it changes IP addresses, it uses another. The only way to refresh is to reboot the box. There is no testing to make sure a connection is still being used.

We installed the 4420 SSL VPN into our system in the spring of 2005 & it was a nightmare. We had to change our Internal IP configuration to allow room for the VPN DHCP clients - not a huge deal, but I wasn't happy that our VAR didn't explain & allow for this during the FW installation. All along they'd planned for us to use the layer 3 tunnel & this was a known requirement.
SSL VPNs can be tough to configure sometimes, but our setup is pretty standard. Microsoft did throw a wrench into the works last year when they 'secured' Exchange, Outlook, Outlook Web Access & Server 2003, so I forgave some issues due to this. Besides that, I needed file links to the Novell servers & some simple HTTP links, all of which worked fine.

The initial plan of using a layer 3 tunnel changed to individual tunnels for each service due to a flaw in the OS. While initially a problem, I'm actually glad this happened since there were no firewall services built into the VPN's OS at that time & I don't have enough control over the remote PC's, so this plugged a big security hole that I'd worried about - although no one else seemed bothered by this.

During this time, the poor performance on the VPN was thought to be a hardware problem. We RMA'd the box. The replacement worked fine for half a day & then locked up. It had a hardware problem. So we were forced to RMA the replacement box.

I had the impression my problems were the first time the techs had ever seen them. We spent a lot of time working directly through level 2 support & the developers. What is a piddling little company like mine doing on the bleeding edge? It's not a spot I want to be in. I don't have the resources to exist there.

We limped along on the original box for almost 4 months. It was a time of constant frustration for me & all my users. The box worked - mostly - but at the price of constant reboots & reconfigurations. There was an annoying bug in the file browser that didn't let some users, some times see all the files in some directories. Almost impossible to reproduce, development finally fixed this bug in September 2005 by replacing the ActiveX component. They kept it pretty secret, though. There was no mention that the hot fix this came in would fix that problem.

In the end, it basically came down to flaws in the OS making the box too limited for us. We finally got everything resolved by Symantec upgrading the 4420 to a 4460 in Aug05 for free. Unfortunately, the first 4460 that arrived didn't work. Again, we had to replace the replacement!

I was concerned that the difference between the xx20 & xx60 boxes would be a problem when we put them into High Availability/Load Balancing (HA/LB) mode as planned later in the year. I was assured by the VAR & Symantec that it wouldn’t be a problem since we could put the same OS on both & set them in an active-passive mode. We would primarily use the x460 & the x420 would handle overflow or everything, only if the x460 was down. This would also keep to the plan of licensing one box for both FW & VPN.

SGS 3, the OS that would make both boxes do both jobs came out in September 2005. Since my support on the 5420 Firewall ended in November 2005, I needed to get the upgrade done quickly. I really didn't want to. If you've followed the story so far, you'll know that September was the first month when everything was finally working. But, it was part of our original plan & not having redundancy in the system is worrisome. So we decided to go for it.

----------------- to be continued --------------

Stop Whining - Be Grateful!

There's plenty of stress, disappointments & flat out lying to contend with when you're trying to ride herd on a network. I know it & whine about it constantly, but there is a lot to be grateful for too, so I thought I'd list a few.

Bosses: They can make your life miserable, but I can pass the buck up to mine. She has to attend all the meetings where they talk endlessly about all the things they should be doing. I get to stay in the server room & do. Mine lets me set my own hours & never watches the clock on me. If I need an extra hour or day off, it's fine with her. Of course, she expects me to stay late as needed, too.

Help Desk: Those wonderful, front-line people that take all the flack for my mistakes & actually talk to the users. They answer the same questions - often stupid - day after day after day & manage to do so politely. Thank you! I couldn't do it.

Novell: OK, it's dying, but for file services, there still isn't anything else out there with finer control over permissions. I had more trouble getting 6.5 to stabilize than most previous versions (I skipped all the X.0 versions), but now that it is running, a reboot once a month is about all the attention those boxes need. Virtual Office is one of the best kept secrets anyone ever had.

Veritas: Alright, their licensing is a nightmare, but the basic product is pretty good. It's a little quirky at times, but the tech support is pretty responsive when you need them & once set up, you can pretty much live with it. (Gosh, I think my standards have gone down hill…)

3Com: They're not Cisco, but they're not priced like it either. Their middle range switches are solid & pretty quick. Plug them up & let them run. I'm still using some that are 7 years old.

Dell: Not only do they make a solid PC & laptop, but now they sell all kinds of other gear. Finally, a one-stop shop. No more need to price everything all over the place, although I still don't use their servers & switches. Just a personal quirk. I have no evidence they don't do a fine job, but I've been using HP/Compaq servers & 3Com switches happily for years. Why change?

HP Printers: They're not the cheapest, but they seem to be more solid than others we've tried. Their drivers are solid too. (I'm still recovering from some go rounds with Kyocera!)

APC: Rarely seen & always relied on, their UPS's are one of the foundations of the business. We had some remodeling going on & I watched circuits flicker & die as the electricians played their games. Network services stayed up!

Microsoft: Thank you for job security!

Phones, Service & Billing

Phone service is usually so good that I take it for granted. Until I try to make a change, then it's like playing dice with the Devil. No doubt about it, I'm in trouble.

I'm old enough to remember when phone service was like using a MAC; everything worked without a hitch, but you had few options. Now it is like using Unix; so many options that even the provider can't keep them all straight. My phone company, Verizon, has so many different departments they don't even know how to transfer you between them.

The regular residential help line in Maryland is 410-954-6260. Getting through to customer service via this line is almost impossible, no matter what time I've called over the past couple of months. It's a voice menu from Hell that spends minutes leading you through a multitude of choices & finally - you're almost there! - tells you everyone is busy & hangs up. Diabolically sadistic.

Luckily, one service rep gave me a number to call her back at, 410-954-6221. This number skips the voice menu & dumps you directly into a holding pattern for the next available rep, usually just a couple of minutes.

Fifteen years ago or so, we had our phone number changed so we could dial in to the city from our rural home without incurring a long distance charge. This is called a Foreign Exchange & we paid almost double for this privilege, but it was worth it. Then we moved a few miles & our daughter became a teenager. Suddenly it was long distance for her to call her friends. It didn't take too long to see that this wasn't going to work, so we thought we'd take advantage of Verizon's Freedom Plan.

This would require 3 items be done:
1) Change my phone number to a local exchange.
2) Pull my current provider & put all long distance service on Verizon with the Freedom Plan.
3) Move my DSL from the current number to the new one.

"No problem, Sir" was the answer. Except the DSL. That complicated things & the normal 10 business day wait wasn't acceptable. I live an hour from my servers & need access to them.
"Is there any way we can get the transfer of DSL service done quicker?" I got transferred to the DSL department who said they could. They transferred me back to the service rep, except I wound up going to some business service office. It took two more transfers & 30 minutes to get back to a service rep - a different one - who hung up on me.

I tried again a couple more times with varying, but similar results before finally hitting on a good rep who got everything done. Except after 5 days, the DSL wasn't up & we were expecting bad weather, so I braved the service line again. Two hours later, they said the DSL would be up the next day. It was!!!

A month later, I got a bill from my long distance company. I was still paying them for long distance. My Verizon bill arrived & I found I was paying extra for Call Waiting, Voice Mail & Caller ID - free services on the Freedom Plan. But I didn't have the Freedom Plan.

Tips: Call 1-700-555-4141 to verify which long distance service you have.
If you don't have caller ID, 200-200-6969 will let you know the number you are calling from. (Don't put a '1' in front of this number.)

I braved the customer service line again. Again, I had to use the back line because the voice menu said everyone was busy. An hour later, the Freedom Plan was indeed added to my phone line & they even credited me a little for the 'free' services. They couldn't & wouldn't make good on the other costs I'd incurred.

Bottom Line: It took me 6 hours & cost me $65, with 5 days of interrupted DSL service to make that 'simple' move to the Freedom Plan. So much for Verizon's customer service.

Salesmen & Details

I investigate & recommend most of the components of our network; hardware & software. Documentation, demos & reviews only tell part of the story. No matter how hard I try, I rely heavily on the salesman. Someone once said, "The devil is in the details." (Ambrose Bierce?) Whoever it was, they were so right. A good example happened to me out last year as I changed Help Desk work order & inventory systems.

I worked at & managed a help desk at my previous job. I've configured several different systems, so I knew what we needed. We had been using Track-it Standard edition & had upgraded from version 3, when I started with the company, to version 6 at that time. It was a pretty good bang for the buck, but too limited for what we wanted now. We had to patch all of our remote PCs & get current inventories, something that Track-it just couldn't do in our environment. About 75% of the PC's we support are in small offices scattered around the US. They're not members of our domain, can come & go as owners change & are used by people who are not picked for their computer skills.

After a lot of research & pricing, I decided on Everdream, an ASP that would do an inventory, patching, help desk work orders & even allow us to push down custom software to our remote PC's. Best of all, they did all this for the low price of $3 per PC per month.

Everdream turned out to indeed be a dream to setup. We had to install an agent on all the PC's & make sure each PC was named uniquely & according to our specifications, but that was expected. It was a lot of work, but the help desk folks got it done. The system wasn't perfect, but was OK. Work orders were a bit of a pain to enter, so we didn't get as many in as I would have liked, but the inventory & basic OS patching were pretty good. Reports were OK. I have to admit that we didn't do as much with the system as we could have, but it was a busy year & it seemed to serve us pretty well - at first.

In March, I needed to make sure all the PC's had the latest version of Java on them for the new SSL VPN to work properly. I tried to roll it out through Everdream, sure that I could, because I'd asked the salesman about this specifically. It was one of the selling points. I couldn't get it to work. Tech support said it would need to be a custom package & it would cost me $2000! No way! So we rolled it out the old fashioned way, by telling the users to install it themselves. How hard is it to go to Java.com & click download, after all? Tougher than you'd think for some of our folks, but everyone managed it.

I started a months long project of getting Everdream to live up to their promise to allow me to roll out customized programs & patches to my sites. Finally, in September, they showed me, but at the same time they informed me that their price per PC would go up to $11 - $12 per PC per month the following year AND the work order module would not be included. We negotiated that back down to $4.50 per PC with the work orders.

The huge price increase coupled with Microsoft finally getting their Windows update working well made me reevaluate Everdream. They made some nice promises, but I wasn’t happy with them. I don't like promises that take 6 months - half the contracted year - to come true. I don't have that kind of time to waste & don't need more stress fighting them for it.

We decided to go back to Track-it, upgrading to 7.0 enterprise this year. Their tech support is quick to answer & pretty good, but even their salesman has made promises they can't keep. He told me that I could turn the Solution Library on or off for my users. Turns out, without 'Service Plus' it is always on. Since we use the Solutions to store stuff for the Help Desk technicians, I really don't want the users seeing it. It turned out the fix (not supported by tech support, who did NOT tell me where to look!) was to edit 4 of the .asp files for the user web client & remove the references to 'Solutions'. My techs can still get to the Solutions through the technician client, though.

So that's the moral of today's gripe. Salesmen make promises their product can't keep & they shouldn't. I know they aren't techs, but tell me honestly what your product can do or make damn sure your tech support is good enough to make it work anyway, even if they're not supposed to.

Administration & much, much more...

Network Administrator seems to be a pretty catch-all phrase these days. Look at the job postings & I see everything from jobs for Help Desk to System Engineering listed under that title. That pretty much describes my job for a small, privately owned company that’s suddenly midsized. I spend long, stressful hours trying to keep the network & it's users under control & headed down the right path.

Budgets, vendors, engineering & more:
I spend a lot of my time now thinking about what we need & setting up new equipment - or modifying existing equipment to meet new needs. New equipment & software seems to be getting worse to set up. Is it just the complexity of the systems, my imagination (ignorance?) or am I really getting beta products? It's a mixed bag, but I intend to write down my opinion on some of those I deal with.

Security:
Threats are getting smarter & so are some users. Too many are still burying their heads in the sand & hoping they don't have know how to operate their magic boxes - that's what tech people are for. We're supposed to make sure they can't make stupid mistakes. Others are quite happy to reconfigure their computer at the drop of a hat.

Administration & Management:
New & terminated user setups are generally done by the Help Desk staff. Are they doing it right? Did they read or fill out the templates on those procedures - have I bothered to update them correctly this year? Last year?

Clairvoyance:
I try to read a lot of articles on what’s hot & figure out what I think the company wants & needs. Not just this year, but over the next few, and then comes the really hard part – getting the suits to go along with it.

It’s all in a day’s work & some days are better than others. Hopefully, I’ll keep track of some of it here.