NO CARRIER

The animation of Steve Scott. Wow.

BIRDY NAM NAM – THE PARACHUTE ENDING from Steve Scott on Vimeo.

kashmir from Steve Scott on Vimeo.

I came across this on ioerror’s twitter. And it made me laugh really hard.

0-day?

Well. It has been a long day. I awoke this morning dealing with a major network outage. Once that passed, I settled into my normal routine of fixing Linux problems. While dumping some 80G of databases, I figured I would get a head start on my migration. I am moving blacknode.net to slicehost, so my first order of work was cleaning up and migrating my zones to slicehost’s nameservers. No big deal there.

Now here is where the trouble starts. For some reason I cannot recall, I used register.com as blacknode’s registrar 3 years ago when I sniped the domain from some German quake 2 gaming clan. After a few minutes of remembering my login credentials, I was there, looking at there root nameserver editing tool. I double checked my work and entered the new nameservers. I clicked submit.

I have walked customers through this process THOUSANDS of times. webmasters and ecommerce noobs have a stigma attached to all things DNS. The words TTL and root NS records seem to go hand in hand with 24-48 hours of downtime. The average domain owner also seems dread making changes at the registrar level; especially when register.com is involved. After todays experience, I think I finally understand why.

For some reason my dns changes would not stick. The interface simply said that the change failed and that I should try again. I did so, and still no go. The register.com website gloats about its amazing call center support. I think this is how they justify being the most expensive registrar on the planet.

So after the receptionist, who politely told me my estimated hold time (5 minutes), I was connected to a techie type who seemed to know what this crazy DNS thing was, or at least new what words to throw around. She went through the dns change process that I had just went through and verified that my dns servers we up and answering queries but ran into the same problems I did. She got me into a ticket and escalated the problem to a tech with access to the backend.

This was around 3:30pm.

I checked the status of my ticket by querying the whois database for register.com. After about an hour, I noticed something very wrong:

DNS Servers:

dns234.c.register.com
dns249.d.register.com
dns073.b.register.com
dns134.a.register.com

It seems that the annonymous technician was able to reset my domains root nameserver record to the default for register.com. I hoped this was a temporary thing, so I waited for about 5 minutes before getting worried. I attempted to alter the record to point to the slicehost nameservers again, but no dice. I tried to put it back to what I had initially, fail.

You see, I migrated my zone from Rackspace nameservers. The zone happily resolves on both slicehost dns and rackspace dns. So not being able to change it was not a problem, as long as the the records stayed what it had been for the last 3 years. It was about 4:30.

So I called up again. Same procedure, I had a ticket number, but the techie still had to jump through the hoops. I was ensured that my issue was being addressed by the highest escalation point available and that they would get to the bottom of the issue. They seemed to believe that the problem existed upstream somewhere, unfortunaitly, those guys live on the east coast, and are 9-5.

While on hold, I began adding the important records to the register.com dns servers. It seems that the original ip I had hosted blacknode.net on was still present on register’s nameservers with a TTL of about 14400 seconds. The ip was a dynamic address used on a residential broadband network. So i ended up caching this on my local dns…

So I was effectively down. Not that this cheesy blog gets allot of traffic. Or like I make money off this site. But if I did, I am sure I would be pissed.

The techie got back on the phone after about 15 minutes and said that they could not resolve the issue without the real technicians who only worked daywalker hours.

I think I will be transferring my domain elsewhere. Godaddy seems to be it.

~

My friend translates and says a scientist has genetically altered these fish to have faces that resemble primates.

I really wish I was able to goto comiccon this year… oh well. I am missing Defcon and Black Hat now. There is alway next year.

SDCC 09

Sometimes the most simple themes are the most entertaining.

Well, the time has finally come. I have had this server for about 3 years now and it has only been rebooted once for a kernel upgrade. I have just been told that the bread rack blacknode sits on has been slated for demolition to make way for cabinet expansion in its San Antonio DC. I really cannot complain about this as I only pay $5/m for a dedicated machine and some bandwidth.

I am still going to miss her when she goes and am going to work on getting her moved beneath my desk for nostalgia’s sake. Anyway, she is not much in terms of hardware, but she has been rock solid in terms of stability. The server hosts this blog, datacenteroutage.com and my svn repositories. It has been a good test bed for anything that does not require allot of memory or processing horse power.

Here is what she looks like:

blacknode ~ # cat /etc/gentoo-release
Gentoo Base System release 1.12.11.1

blacknode ~ # uname -a
Linux blacknode.net 2.6.25-gentoo-r7 #4 SMP Tue Nov 11 01:57:56 CST 2008 i686 AMD Athlon(TM) XP 2800+ AuthenticAMD GNU/Linux

blacknode ~ # cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : AMD Athlon(TM) XP 2800+
stepping : 0
cpu MHz : 1658.561
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow up ts
bogomips : 3320.27
clflush size : 32

blacknode ~ # cat /proc/meminfo
MemTotal: 513280 kB
MemFree: 31688 kB
Buffers: 155724 kB
Cached: 91216 kB
SwapCached: 1488 kB
Active: 288128 kB
Inactive: 110332 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 513280 kB
LowFree: 31688 kB
SwapTotal: 995988 kB
SwapFree: 991280 kB
Dirty: 132 kB
Writeback: 0 kB
AnonPages: 150868 kB
Mapped: 45676 kB
Slab: 76932 kB
SReclaimable: 67864 kB
SUnreclaim: 9068 kB
PageTables: 1824 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1252628 kB
Committed_AS: 531644 kB
VmallocTotal: 507896 kB
VmallocUsed: 2668 kB
VmallocChunk: 505224 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB

blacknode ~ # lspci
00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge (rev 80)
00:01.0 PCI bridge: VIA Technologies, Inc. VT8237/VX700 PCI Bridge
00:0a.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)
00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South]
00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
00:11.6 Communication controller: VIA Technologies, Inc. AC’97 Modem Controller (rev 80)
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

blacknode ~ # uptime
16:21:52 up 247 days, 1:30, 1 user, load average: 0.24, 0.08, 0.03

Only one little hiccup:

MCE: The hardware reports a non fatal, correctable incident occurred on CPU 0.
Bank 2: 940040000000017a

I am going to be migrating to one of my other, more powerful environments in Dallas Texas. I am still a little sad.

In an article here, Rackspace is announcing the availability of their beta web services api that can be used to jack into their cloud. Much like EC2, this should lead the way for projects like scaler, that dynamically grow a configuration as thresholds are reached.

Check out http://www.rackspacecloud.com/ for more info. I am going to sign up, I will let you know how it goes.

I am working on a new blog.