lame sysadmin

Weird stuff went on recently on my network;

It started last night while I was fiddling with my lab network - dark witchcraft including ospf, eigrp, nat, acls, ppp/chap, etc. I discarded it as being my laptop still fussing with its gateways and default routes... I was tired and didn't care. So I just turned things off, went to bed and forgot about it.

[side note: I should post my network diagram someday, so everyone can appreciate a true overkill geek home network...]

Cisco home and lab networks, originally uploaded by pjvenda.

This morning, though, my wife couldn't reach facebook from her laptop. Whoops, I must have done *something*...

Symptoms were:

• able to ping external hosts;
• unable to access websites;

First things first: Check the squid proxy - seemed ok;

Lazy as I am, I decided to just reboot the routers because I might have left some not-very-well-though-configs there. The routers came back up and the situation got worse! Now I had no comms between the core and uplink routers. WTF?

Moral of the story #1

Having bpduguard on switch ports that link to routers is not necessarily safe.

The router was broadcasting BPDUs out its ports triggering bpduguard on the switch which, in turn, disabled all the ports that connect to the router (the ones from which BPDUs were received). This effectively shut the router off the switch.

As a quick & dirty solution, I enabled bpdufilter on those switch ports. Just ignore those BPDUs instead of disabling the ports if one is received (bpduguard is setup globally on the switch, rather than port by port); The proper solution involves shutting down STP on the router instead with no spanning-tree vlan X commands.

Layer 2 issues sorted, I was back to last night's situation. Being a little more pragmatic this time, I disabled wccp redirects from the core router. Sure enough, everything was back to normal. The proxy was good but the connection between the core router and the proxy service/host was not (this connection being a GRE tunnel - did I mention how nice WCCP is??).

Moral of the story #2

Be pragmatic, not lazy.

I had a massive update pending on the server including a kernel upgrade. I had to reboot now. Naturally the server did not come back up as expected because the updated version of udev required a kernel >=2.6.27. The latest accessible kernel was 2.6.26 - d'ough!!

So there I was manually creating md nodes on /dev via a serial console to mount and copy the newest available kernel, modify grub's configuration and try again with a new kernel. It worked! Everything came back to normal, the GRE tunnel came back up, as all other services on the server.

Moral of the story #3

Don't slack on your sysadmin duties, update often and check that things are still working after updating. Make sure you reboot from time to time to verify that everything is starting up nicely.

Along with all this, I forgot about the new terms of service of editdns.net (my DNS provider) and they cancelled my account. My domain pjvenda.net has been unavailable since 3rd January 2010 and should only be restored after tonight (10th January 2010).

Moral of the story #4

Make sure you have a working DNS service for your domains. Otherwise they don't work.

I'm lame sometimes. Must be the cold.

Cheers, PJ

Photographic record of the last ride of 2009

I know there is still time, but I don't expect many more convenient breaks in this British weather.



I took this picture after coming back from a short ride earlier this month - an English December, mind you. It was a good ride through Oxford city streets. The weather was dry and not too cold (for my kit). If everything seems grey and dull, it's because it was.

I doubt I'll be able to go out again this year and if so, here it is: the record of the last ride of 2009.

Cheers, PJ.

Night lights at the Colosseum

Night lights at the Colosseum, originally uploaded by pjvenda.

Taken from the hotel's balcony, while having dinner, during my honeymoon.

The balcony shook as every tram or bus passed by, but the result was not bad. 30sec exposure done with an EOS 350D + Sigma 18-125mm.

Posted one day after I signed up for a flickr Pro account.

Back to base

Back to base, originally uploaded by pjvenda.

At Farnborough 2008.

Elevator shaft

Elevator shaft, originally uploaded by pjvenda.

Going up?

CSS is magic

90% of the effort required to change this:

into this:

(which is awfully similar to this:)

was invested on a Cascade Style Sheet.
HTML+CSS can be about as good as LaTeX.

Hi-Tech

Hi-Tech, originally uploaded by pjvenda.

Moto Guzzi Griso 8V. A clever looking pure italian bike.

This is the first post in a "photo blog" style that I may start to introduce on this blog as well.

Cheers, PJ.

Lead-acid battery maintenance and diagnostics: What I have been doing

The stuff you learn as you go along ...
... needs to be shared with those that might not know yet.

When my Virago would not start two weekends ago, I and my friend jump-started it off his own bike. Once warm it would start again normally, so we set off for a 120 mile ride and I attributed the problem to myself: "I just flooded it when I first pushed "start" but now the alternator should get things in place again..." But that did not happen. Since then the bike did not start again without help, that is. It only fired up again with the help of the car's battery. I could jump-start it from the car, go for my ride and hope that it would start again (while warm).

Troubleshooting begins:

• Starter cranks the engine a bit (half-turns), neutral light fades while it happens
• The headlamp still works with the bike off

Seems obvious - it's a weak or half-dead battery.

Immediately I ordered a fancy charger (yes, it's blue! but in my defense, it was not my choice) to check and charge the battery, which failed miserably because it has not been delivered yet... Seems to be out of stock everywhere, I should have known better... Well, before the charger comes in, I figured I would do what I could to get more information out of the situation.

The exact model of the battery is Yuasa Yumicron YB12AL-A2. It's a lead-antimony type 12V 12Ah motorcycle battery. Yuasa kindly provide this very insightful document in their website summarising the technologies behind their ranges of batteries and how they fit into different types of requirements.

Before even taking it out, I noticed that no fluid could be seen around the MAX or MIN levels. Either because the plastic was too opaque or because the level was not even close to those levels. Next thing was measuring voltage levels in different situations: no load, loaded but engine not running, engine running. Results were approximately:

• Unloaded: ~13V
• Loaded: N/A (didn't measure)
• Running at idle: ~18V
• Running over idle, unloaded: ~17V

Aside from potential regulator/rectifier issues, I saw nothing particularly alarming here. So I took the battery out to have a good look at it. This visual inspection revealed the following:

1. All connectors and metallic contacts were in very good condition - no corrosion or grime even
2. The battery itself was clean and looks recent
3. There was no fluid whatsoever inside it. It was completely dry

All sorts of trouble may arise from semi-dry batteries, including quicker corrosion of internal and external materials due to increased acid concentration and electrical properties must change as well - from internal resistance, maximum current output, voltages, etc - nothing tolerable in the medium/long term.

What I cannot understand is how a completely dry battery could conduct *any* current, as it did. I would expect a battery to fail gradually before being completely dry which was not the case. A bit of misleading research led me to believe that this particular model of battery soaked the fluid (into some material) which would explain why it would not drip... True or false I don't know yet, as I could not find that information again. Oh, and there is also the little detail about: "Is the dry battery a cause or a symptom?" - I do not know yet.

If the battery was dead I would need to replace it. But it could also be brought back to life by filling it with de-ionised water and recharging. Having nothing to loose, I bought one liter of de-ionised water and (carefully) poured it in. [bloody filling holes must be made to be filled with seringes!!]

Immediately I measured a drop in voltage of about 1V. Back on the bike, I re-measured its voltage just to double-check that things were OK. Turned it on, re-checked things quickly, hit START and it fired-up promptly! Once more I measured voltages with the engine off and on, empty and loaded (lights on).

• Unloaded: ~12.1V
• Loaded: ~11.75V
• Running at idle, unloaded: ~17V
• Running at idle, loaded: ~13.5V (~15.5V when a little warmer, may go up a bit more)
• Running over idle, loaded or unloaded: ~16.5V

With the engine running, the electrical system should be mostly (or totally) powered by the alternator (stator/generator) through the regulator/rectifier, while at the same time recharging the battery for the next start-up (voltage at battery terminals should be >12V). This makes sense and my measurements are consistent. However, three things are still concerning me:

1. Loaded voltage is slightly below 12V and seems to drop quickly
2. Power coming from the charging system may have a higher than normal voltage (~17V)
3. According to manufacturer specs, 12.1V of unloaded voltage is equivalent to about 50% of charge

Best case scenario:

Battery is still good just needs maintenance. The least of my worries is buying a new battery.

Worst case scenario

Either the stator/generator or the regulator/rectifier (or both) systems are frying the battery. Not nice as the following might have to be replaced:

• Regulator rectifier
• Stator - Generator

While at it, why not read a bit more about the whole charging system. Knowledge is power.

• Wikipedia: Lead Acid Batteries article
• electrosport's How motorcycle chargins system works
• wemoto's Regulator Rectifier technical description and quick troubleshooting
• wemoto's Battery Care Guide
• Bit of theory with schematics and photos of regulator/rectifiers

Bye for now.

Home network / Cisco testing lab

[1]: "Dog's bollox' syndrome", autistic geekiness, paranoid overkill network admin, will to learn. Call it what you like, I don't care.

This is about my home network. I've been getting my hands on some Cisco kit because it has always been a technology I wanted to learn about. After gathering a few newer or older devices, I was able to re-do my home network, with separated user, management, service and DMZ networks - done with the good help of ACLs, NATting and inter-VLAN routing. Internal services are explicitly accessible or transparently proxied, external service are NATted into the DMZ network. I am even setting up a direct Internet access network as fall back. Most of this stuff is provided over Wired Ethernet and 802.11g Wireless. Why? See [1] (top of post).

It all looks like this at the moment:

Cisco home network

At the moment, the following devices compose and actively participate in this horrible wire mesh that I call "home network":

• Cisco Catalyst 3548XL: 48 10/100Mbps Ethernet port core switch;
• Cisco 2621XM: Core/Internal router and firewall;
• Cisco Aironet 1231G: Wireless access point;
• Cisco 877W: External router and firewall with DSL modem and wireless interface;
• Linux server: (not seen in the picture) Provides storage and other network services such as SMB, authoritative and cache DNS, HTTP proxy, etc; There is a web page about this box on my website: Home server;

Because I am still a Cisco "newb", I needed to learn about the features that I needed as I went along. Stuff like configuring native and tagged VLANs with trunks on the switch, multiple networks on the access points (each going to a different LAN segment), setting up static routing, ACLs and NATting, enabling and using an ADSL cisco interface, managing Cisco IOS images and configurations, etc. It starts making a lot of sense as all the pieces of the jigsaw find their place in my mind, but it seemed a bit strange when I started.

I know for a fact (because I have seen them) that certain company setups are light-years away of mine both in size and in complexity. What is funny is that certain cases are light-years more complex but others are light-years simpler (d'ough!).

I quickly learnt that a network like this requires absolutely the permanent presence of an up-to-date network diagram and detailed notes with IP addresses and ranges, interface names, VLAN to IP mappings on the various devices, VLAN and trunk mappings on the switch.

[A random network diagram I ripped off the Internet for illustration purposes only. Mine is hand-written.]

I also learnt that Cisco is a huge corporation in such a good market position that allows charging of hefty sums for hardware and licensing together with software limitations on their devices (dependent on licensing levels). This is not easy for me to swallow. But I must say that their documentation is terrific in every aspect! In particular the following guides are absolutely top-notch in content and references:

• Configuring IP Addressing
• How NAT works
• NAT Order of Operation
• Configuring IP Access Lists
• Cisco Guide to Harden Cisco IOS Devices

There are many more great Cisco guides that should be on the list above, but it would just group exponentially. And I am not even thinking about Cisco Press paper books.

Apart from being an over-zealous sysadmin, I am also a bit paranoid with security. So I will not be giving away details about configurations :]. At least not at the moment.

Following the logical steps, I should be taking a CCNA exam in the near future...

Cheers, PJ.

Website is back with a few changes

After about a month offline, I managed to bring my website back. It's now hosted in Canada attached to a nice big 100Mbit/s network pipe.

It's also been slightly updated with a nice little flickr photo of my collection shown in every page.

Other changes include a new domain registrar, DNS system and domain name to which my stuff will slowly be ported to. I will maintain the old domain indefinitely but things should redirect to the new one. So have a look:

http://www.pjvenda.net

This blog is now accessed at http://blog.pjvenda.net. The old address transparently redirects here, so no worries. I plan on integrating it with the website but that may take a while.

See you later!
Cheers, PJ.