I hadn’t heard anything else from Peter until about 5pm when he wrote to tell me that there were now 3 technicians working on the array and they will be performing a ‘syswipe’ to resuscitate the array. A syswipe is where they completely reformat (wipe) the operating system, reload it and rebuild the configuration. They said that the 7 terabytes of data would be fine.

When arriving on the scene at about 7pm that Wednesday, I learned that the array said it was happy, but the Citrix Xenserver couldn’t see its 5 storage repositories that get served up from the array. We worked on trying to get the xenserver to talk to the array for about another 3 hours. We could get it to see the LUNs (logical unit numbers), but when it looked for the data, it stated that there was no data present. In fact, manually using fdisk, we didn’t even see partitions. It was like we were starting with fresh, blank volumes. We left for the evening with Dell scouring the support information to see if they could find any flaw with the way the syswipe was performed and they would call when they came to a determination.

At about 2am, Dell called and we conferenced myself, Peter, and about 3 or 4 people from Dell’s side. The news wasn’t good. They couldn’t find anything wrong with disk alignment or configuration that would lead them to believe that they did anything wrong and they were convinced that the xenserver was to blame for the missing data. I was dumbfounded. I couldn’t believe what I was hearing. They basically told me that this was now a Citrix support issue and they were basically done. Trying to stay calm, we decided to hang it up for the evening and get some rest.
Thursday was a bust for continuing on recovery with Peter and myself, but on Friday, I started to research the log of commands that Dell performed on the array for the syswipe. From the documentation I found, there are 2 types of commands you can perform: create commands, and recover commands. Guess what Dell ran. Both. On the inside, I was getting pretty steamed. Convinced that Dell blew away the data, I was formulating my argument of what Dell was going to do to rectify the situation.

Two days ago, we had an error on our array saying that we lost some redundant capability and were up and running and now we were at complete dataloss at the hands of Dell. Peter had a feeling that we weren’t going to be able to recover the data, but I *KNEW* the data was GONE.

At this point, we both let the project chill for about a day. We started talking about possible scenarios for data recovery with going back to the old array we had just replaced, but since the new backup procedure wasn’t in place yet, there would certainly be things missing.

And then came Sunday. It was Hope’s 16th birthday! it was a time to celebrate. We even had cupcakes : ) Crissy was helping in my 5yr old’s Sunday school class and my 3yr requested to go play so at sat by myself at the 9:15 service that day. Up to now, I have been the strong one in process up until we started to sing. I was a mess. Thank goodness I wasn’t sitting near anyone I knew! I wasn’t crying, but my eyes were rather moist. I kept telling myself “For crap’s sake, it is *just* data.” There are way bigger problems in this world! Here is the thing, I wasn’t crying because of dataloss. I was crying because I *knew* something was going to happen. I said (in my head, of course) “Damn-it, God you are going to freaking fix this problem and I won’t be able to ignore it and you anymore.” Let me make this clear. I wasn’t demanding that God fix it. I knew that it was about to be fixed and I’d have to talk about it. Still in denial, had a couple conversations on the way out with Chris G. (operations director) and Peter about next steps and went home.

After lunch, I went upstairs and started running the same commands I ran over and over trying to introduce the storage repositories to the xenserver with the same failed results. While sitting there, a single word came to me: IPV6. I literally said out loud “What do you mean IPV6?” I don’t know where it came from or what it meant, but there it was: IPV6. How could that have anything to do with anything. I was doing explicit single pathing on a particular IPV4 IQN just like when I first configured the array. Frustrated, I turned off my computer and went back downstairs to get ready to take my family to go see Megamind. As we were about to go out the door, I had a spark of an idea. I called Peter and asked him to look at the array and if he saw IPV6 on any of the iSCSI interfaces, completely disable it. I told him I was taking the kids to a movie and I would check on it when I got back.

After the movie, I went back upstairs and tried the same procedure that I had performed NUMEROUS times. It.freaking.worked! I successfully setup all 5 storage repositories and re-linked all the virtual hard drives back to the storage server! I turned on the server and it was like nothing had happened. Everything was back to normal. I started weeping like a baby. Silently of course, I am a man : ) I called Peter and we both sat dumbfounded on the phone in disbelief. After I got off the phone, Crissy walked in and asked why I was crying. All I could tell her is “Oh nothing, I’m just really happy.” It was certainly NOT nothing.

To top off the story, on Monday, Peter heard back from Dell. They had reviewed all the support calls again and determined that due to everything that had happened, our data is completely scrambled that no data recovery company would be able to retrieve any data. I bet they were a little surprised when he told them that everything was back to normal.

Dell, I’m sorry for all the bad things that I said to you in my mind. I certainly didn’t approach you with a loving heart.

If I could please hold your attention a little bit longer, I want to point you to the sermon from that Sunday. It is about 45 minutes long, but worthy of your time. Thanks! Click here to play it.

Luckily, a couple weeks ago, I learned about launchctl. It is now the primary way in OSX to launch system daemons instead of init scripts or xinetd. It turns out, using launchctl, as root, you can execute a program in another context using its bsexec subcommand! If you use the loginwindow process id’s context you’d like to run your program in, it will now run just as if you were physically sitting at the keyboard.  The command:

isightcapture image.jpg

which would before fail if you ssh’d into a remote machine now becomes:

export PID=`ps auxww |grep loginwindow |grep -v grep |awk ‘{print $2}’`
launchctl bsexec $PID isightcapture image.jpg

Bamo! Now you’re breakin’ the law in style!

So, since I’ve got my crappy little image uploader working once again, I thought I’d share it with the rest of the world. I’m using sleepwatcher, isightcapture, and some other craziness that I hacked together to overlay some good information on a picture from your webcam and upload it to a website of your choice!

If you know what you’re doing, you should be able to follow my instructions just fine, if you don’t know what you’re doing, you might possibly blow up your mac and kill a kitten.  You have been warned!

First, download the following file (I’m going to assume it will download into your Downloads folder):

http://tompohl.com/files/locateme.tgz

Create a directory on your web server to host the supporting scripts and uploaded images for example:

mkdir /var/www/html/locateme

There are 10 types of people in this world.  Those who understand binary and those who don’t.

Let’s talk about numbering systems.  Specifically Decimal (base10), Hexadecimal (base16) and Binary (base2).

First off, what is this base stuff that you’re talking about.  The base number in a numbering system is the number of values represented by a single digit.

Decimal – Base 10

This is the numbering system that everyone should know.  Each digit can have up to 10 values 0,1,2,3,4,5,6,7,8,9.  You might be saying “Wait a minute! Why is it called base 10 if its highest number for a digit is 9?” I’d say, fantastic question! Here’s the secret: nothing is really something. For each additional digit added in a number, you increase the total number of values that you can represent by a factor of 10. That is:

1 digit  (0 to 9) = 10 values =  10 = 10^1

2 digits (00 to 99) = 100 values = 10 x 10 = 10^2

3 digits (000 to 999) = 1000 values = 10 x 10 x 10 = 10^3

Now, don’t get freaked out when I say 3 digits can be 000 to 999 because 000 is the same as 0 is the same as 0^3. Just don’t confuse it with 0^0 which would be 1 (I already told you, nothing is really something).

So, if you take the base of your numbering system (10 in this case) and take it to the power of the number of digits you’re using, you’ll get the total number of values you can represent with that number.  Note: the highest number you can represent is 1 less than the total number of values you could represent because nothing is really something!

Example math:

 1  (overflow aka carry the 1)
  9
+ 1
---
 10

Hexadecimal – Base 16

I’ve included hexadecimal in this discussion simply because it’s cool and you’ll find it in a lot of places on the web.  A single digit can represent up to 16 different values 0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F. That’s right, you just got your alphabet in my numbers and I just got my numbers in your alphabet! In this notation, A = 10, B = 11, and so forth up to F = 15. A lot of times, when you see a number that is represented in hex (the short name for hexadecimal) they might tip you off by placing a 0x before the value.  This is important BECAUSE once we get into multiple digits, a hex number without any letters looks just like a decimal number, but its value totally different!

1 digit (0 to F) = 16 values = 16 = 16^1

2 digits (00 to FF) = 256 values = 16 x 16 = 16^2

3 digits (000 to FFF) = 4096 values = 16 x 16 x 16 = 16^3

Example math (15 + 1 = 16 right? Here’s how you do it in hex):

 1  (overflow aka carry the 1)
  F
+ 1
---
 10

Binary – Base 2

OK now the whole reason for this entry. We’re talking binary 1′s and 0′s the language of LOVEly computers. In binary, there are only 2 values, 0 and 1, off and on. I’m only going to deal with unsigned binary numbers (no negative numbers) because in my future entries, I’ll be explaining things like masks and bytes in regards to TCP/IP networking. Besides, I’m a generally positive guy and try to not dwell on the negative. On to the numbers!

1 digit (0 and 1) = 2 values = 2 = 2^1

2 digits (00 through 11) = 4 values = 2 x 2 = 2^2

3 digits (000 through 111) = 8 values = 2 x 2 x 2 = 2^3

8 digits (00000000 through 11111111) = 256 values = 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 2^8

Example math (1 + 1 = 2 of course! Now, let’s see what that looks like in binary):

 1  (overflow aka carry the 1)
  1
+ 1
---
 10

Ahh yes! There it is! The punch line to our joke! Feel free to laugh OR cry OR just pitty me for regurgitating this stuff from memory.

Bring on your questions! Do you need more examples? You might be asking yourself, “But in the world does this have to do with the OpenVPN configuration that Tom’s working towards?” For that, you’ll have to keep coming back to find out!

Actually, a couple people have asked me to write up some instructions on getting remote access to a home / office network so they can use remote resources like Apple’s Time Capsule or iTunes music shares.

I’ve decided to rise to the challenge, BUT I feel that I need to lay a solid base of knowledge before I tackle explaining my solution.  My goal is to help you make smart decisions about designing your network.

Over the next several posts, I plan to cover the following topics:

1. Numbering systems (Decimal, Binary, Hexadecimal)
2. Bits, Nibbles, Bytes, Words
3. TCP/IP Networking
4. Using OpenVPN to access your network remotely

To some / most of you, a lot of this will be boring or old news to you, but please keep with it! If anything, correct where I’m wrong, call me out when you think I’m full of it, and most certainly, ask for clarification when you just don’t understand what I’m talking about.

Let’s DO this!