Software engineering in a medium or large organization, especially for an existing application, is a lot different than programming at a startup or on my own. It’s not inherently worse, but it can be frustrating if you walk in the door expecting to spend most of your time writing code.

Startup

1. Idea: Wouldn’t it be nice if…
2. Design: You keep the idea in the back of your brain for a day or two. (You could do it immediately, but in my experience it’s better to wait a bit before implementing a great idea, because frequently you will realize there’s a better way.)
3. Implementation: Code it up
4. Test: Make sure your change didn’t break anything

Total time: 4 hours

Medium/Large Organization

1. Idea: Wouldn’t it be nice if…
2. Track: File a bug to track the issue
3. Approval: Get approval to include the feature in the current release
4. Design Review: Write up the design and implementation of the proposed change
5. Engineering Signoffs: Discuss change with co-workers and get signoffs as needed
6. Cross-department Signoffs: If the feature impacts other groups, get their signoff. (If it doesn’t, you may need to get their signoff anyway to confirm they agree it doesn’t affect them.)
7. Implemention: Code it up
8. Engineering Test: Make sure your change didn’t break anything
9. Internal Docs: Update internal documentation, release notes, upgrade procedures, etc
10. Discuss with QA/QE: So that they can test your change
11. QA/QE Test: QA/QE should test every new feature
12. External Docs: Work with doc writer to update customer documentation

Total time: 2-3 days, assuming things go smoothly

It’s important to note that none of the steps required in a Medium/Large Organization are unnecessary or even overly bureaucratic. Any change to the code has the potential to impact a lot of people and groups. For an existing application, no change is better than a bad change, and even a good change isn’t that useful unless everyone else knows how to take advantage of it. Going through a change control process makes it much more likely that all your changes will be good changes.

The rest can be seen at http://photo.oscarc.net/gallery/5327933_yPpXu.

Don’t underestimate the Summer Comiket!

Haruka and Yuuto go to Dragon*Con Anime ExpoSummer Comiket. Like most big-convention newbies, they underestimate the lines, the crowds, the aggressive photographers, and the general atmosphere.

Watching this episode reminded me that Dragon*Con is this weekend. Having already gone on two vacations this year, attending Dragon*Con this year wasn’t really an option. Even if I had the vacation time, bringing Mai to Dragon*Con would have been difficult, and I didn’t want to abandon my wife so I could go myself.

The constant anime references in the series continued, though it seems like at least half of them in this episode were to Shana. The inside jokes — Tomadoi Bitter Tune, Cat Bus Stop — were amusing. Not that any of this is unexpected; this is an anime about Otaku, after all.

Overall, a fun, light-hearted episode that highlighted the growing bond between the two (and Haruka’s innocence). Based on the ominous-looking goon at the convention, the converstation with the Mika afterwards, and the cliffhanger-ending, it looks like things are about to get more depressing.

Photos from our week in San Francisco are now up:

More at http://photo.oscarc.net/gallery/5781655_WWQpZ.

Rackspace held a sysadmin contest at LinuxWorld. I won first place, and got the top prize, an iPod Touch. I’ve never belonged to the Cult of Apple, and have never been particularly fond of iPods. They are expensive relative to their competitors, and I hate the iPod scrollwheel UI. Also, I think iTunes, especially the syncing UI, is awful. Still, my previous MP3 player (from Toshiba) died not too long ago, and I wasn’t going to turn down a free $299 MP3 player.

The first thing I tried to do was upgrade its OS, since it came with 1.x built-in. I paid my $10, downloaded the upgrade, and tried to install it. The install failed, leaving my iPod in recovery mode. Over the next week and a half, I spent hours reading forums, waiting at the Genius Bar, and trying various things, I was finally able to upgrade a replacment unit to 2.0.1. At last, I could play with my new iPod Touch.

I quickly came to the conclusion that the iPod Touch is not really an MP3 player. It’s a full-blown PDA whose marketing emphasizes music and video. It can do everything Palm’s last real PDA, the Tungsten TX, can do:

• sync calendar/contacts/email
• surf the web using WiFi
• play music & videos
• view photos
• play games
• install third-party applications

The iPod Touch has all the improvements Palm should have included in the TX successor:

• More colorful, friendly, and polished UI
• Enough internal flash memory to store a decent amount of music and videos
• A better web browser
• Easy music, photo, & video syncing
• A higher-resolution screen
• Easy over-the-air application, music, and video downloads

There are a few things I miss from the TX, but not much:

• Ability to install applications from anyone I want. (For the iTouch, Apple has to approve each application, and only allows installation from the iTunes store.)
• Page up/Page down hard buttons. A must when reading lots of text
• Copy/Paste
• Stylus support for precision tapping without fingerprints on the screen. Stylus support would also make working with the onscreen keyboard much less frustrating.

When I was at Palm there were a number of people, including me, who advocated taking an existing Treo design (the 650 or 700), replacing the cellular radio with a WiFi radio, and releasing it as a PDA. Palm Product Marketing at the time emphatically stated that there was no market for such a device, PDAs were dead, and building a WiFi Treo would be a waste of time. The iPod Touch proves that they were wrong; one small item in a long list of missteps by Palm in the last few years.

Surprising absolutely no one, Shiina is now in the same class as Yuuto and Haruka.

This episode seemed to go out of its way to shatter anime cliches:

• Male lead walks in on a half-naked girl. It was his fault, even, but the girl doesn’t throw a megaton punch (or even get mad).
• Character trains hard for a big competition. Not only did she not win, she wasn’t even able to finish.
• Male lead accidentally pulls off a girl’s bathing suit. Not only did she not get mad, she tries to cuddle up against him.
• The obligatory beach episode (BBQ, watermelons, and all) takes place at the school swimming pool
• Male lead offers to help another girl learn swimming; instead of getting jealous, female lead offers to help and shows no serious signs of jealousy.
• Big-boobed women in a skimpy bathing suit tries to catch male lead’s attention. He doesn’t stare or get nervous; he just pretends she’s not there.
• Classmates storm in to find the male lead with a half-naked classmate. Instead of assuming the worst, they completely ignore the male lead and ask how the girl is doing.
• The female lead’s younger sister calls the male lead “onii-san”, not “onii-chan”.
• The male lead is completely ordinary. He’s not perverted, wimpy, stupid, or nerdy.

Other thoughts:

• The secondary characters (the maids, Mika, Ruko, etc) are amusing without being annoying
• Haruka sure is dense!
• Haruka and Yuuto make a very nice couple
• Haruka doesn’t seem to have much to worry about. When she and Shiina came out in their bathing suits, Yuuto stared only at Haruka. Also, the two of them were almost acting like a married couple in this episode, helping out a mutual friend of theirs.
• I hope this map is going to be better than Haruka’s last map.

As mentioned in an earlier post, server virtualization was a hot topic at this year’s LinuxWorld. This post will discuss some of the advantages and disadvantages of virtualization, and the various types of virtualization solutions in use.

What is Virtualization?

On a non-virtualized system, only one operating system (OS) can be running at a time. Virtualization allows a system to run one or more “guest” OSs on top of the “host” OS. Virtualization software tricks the guest OS’s into thinking they are running directly on hardware, when in fact they are running within a Virtual Machine (VM).

For example, a Windows XP system could have a copy of RedHat in a VM, and Windows Server in another VM, assuming the hardware is powerful enough to support having three OS’s running at the same time.

Initially, virtualization was used mostly by engineers for development and QA, because virtualization was a big time saver. For instance, since a guest OS’s entire disk image can be a regular file in the host OS, you can clone VMs easily. Thus, a QA engineer could be guaranteed an exactly identical system each time they ran a regression test. Also, testing a server with multiple OSs became much easier — instead of having 10 physical client systems (Win95, Win98, Win2k, WinXP, MacOS X, etc), a QA engineer could have 10 different VM images on one physical system.

What is Server Virtualization?

Server Virtualization is when VMs are used to host production services, such as external web sites, email servers, file sharing, etc. Server Virtualization is different from development/QA system virtualization in serveral major ways:

1. Performance and reliability are paramount.
2. Server Management is more important, especially cross-server management. If you have 30 physical host systems, the management software must let you view all of their status info at the same time.
3. Each physical host must be able to support a significant number of VMs at a time.
4. The VM software must support high-availability features such as failover, moving VMs from system to system “live”, and load balancing of VMs.

Why Server Virtualization?

Application Isolation

Installing multiple server applications on a single server without virtualization leads to several issues. First, there is the possibility of application conflict. For instance, app A may require a particular patch that app B won’t work with. Second, system downtime has to be approved by all the app owners. If the owner of app A only wants downtime 8pm – midnight, and the owner of app B only wants downtime between 2am – 4am, getting system downtime approval becomes a nightmare. Also, any problems will usually be blamed on the other app. App A is slow? Must be App B’s fault!

Installing each app onto its own system solves this issue, but is wasteful. What are the odds that App A needs even 10% of a modern system’s CPU?

Virtualization solves this issue by giving each app its own OS instance. Each OS instance can have different patches installed, can be brought down independently of the others, and provides isolation from the other applications. However, all the OS instances can share the same hardware, leading to efficient hardware usage. If the underlying Host OS needs to be brought down, the VMs can be migrated “live” to another Host system for the duration of the outage, with no downtime required.

Scaling

Most applications can’t take advantage of more than one or two processor cores, or if they do, performance doesn’t scale very well. By running multiple one-CPU VMs on a multi-core server, with a copy of the application running in each VM, the application can take full advantage of a multi-core system. For instance, running Apache httpd within 8 1-CPU VMs on an 8-core host system will provide better performance than running Apache httpd directly on top of an 8-core server.

Hardware Independance & Fault Tolerance

Any VM can run on top of any hardware, as long as the hardware is running the same virtualization software. Most virtualization solutions allow VMs to be moved from host system to host system “live”, with no interruption to the guest OS, as long as the VM’s disk is on shared storage (such as a NAS or SAN). This has serveral advantages:

If the underlying hardware or host OS needs maintenance, VMs can be moved off of the system beforehand, eliminating any service interruption.

When an application outgrows its current hardware, it can be migrated to more powerful hardware without any downtime, much less any reinstallation and data migration pains.

If a host system fails unexpectedly, any other host system can run the VM, making failure recovery much quicker. In fact, some virtualization solutions allow two host systems to run the same VM in lockstep, so if one host system fails unexpectedly, the other can take over with no service interruption.

Security

By controlling a VM’s access to disk, network, and memory resources, virtualization software can help keep VMs secure. Any virus or root kit that modifies the guest OS to hide itself would still be fully visible to the virtualization software. Also, the guest OS could request that certain memory regions or disk resources be made irrevocably read-only on boot, preventing malware from writing to those regions.

Server Virtualization Challenges

Complexity

The number one downside of virtualization is complexity. Complexity always makes things harder to manage, harder to understand, and harder to troubleshoot. A well-designed virtualization infrastructure manages the complexity by imposing standards and procedures, and documenting everything. A poorly-designed virtualization infrastructure quickly becomes very fragile and impossible to manage.

If a VM is running slow, is it the application? The guest OS? The virtualization software?  The host OS? The host hardware? Shared disk storage? Did the VM move to a different host server? If your virtualization software can automatically move VMs among host servers, do you even know which host server was running the VM when the issue appeared?

More Things Can Go Wrong

The virtualization software is one more thing that needs to be learned, installed, patched, managed, upgraded, and troubleshot. While it would be nice if the virtualization software never had bugs or glitches, that’s certainly not the case.

More OS Instances to Manage

Each OS instance in a VM is one more OS instance that needs management, such as security patches, anti-virus software, etc. If your current patch strategy is to run Windows Update by hand on each system, virtualization will kill you.

Performance Overhead

Virtualization software imposes a performance penalty, especially for disk and network I/O. Also, because each VM is running a copy of the OS, each running VM imposes memory overhead. Full virtualization (described below) has the highest overhead.

There are several ways to mitigate these issues. Using container-based virtualization (described below) or paravirtualization (also described below) reduces overhead. Also, manufacturers are beginning to release virtualization-aware network and disk controllers that speed up I/O from within VMs. Finally, Intel and AMD have added virtualization-specific CPU instructions in their newer CPUs that reduce virtualization’s performance overhead even with full virtualization.

Cost

Purchasing commercial virtualization software is not cheap. If you go with a free solution, you may save on licensing costs, but will need to spend more time implementing the various management tools you would have gotten with the commercial software. Also, server virtualization requires better OS, application, and performance management tools, which you need to either purchase or implement.

Security

It’s possible that the virtualization software or your configuration has a bug that allows hostile software in a VM to “escape” into the host system. Now, it has full control of all the VMs on that host system. Virtualization software vendors take security seriously, so this is relatively unlikely, but…

Virtualization Types

Full virtualization. In full virtualization, the guest OS is completely unaware that it’s running within a VM. This is the most flexible type of virtualization, as it can run any OS unmodified, but it also has the greatest performance hit because the VM has to fully emulate hardware.

Paravirtualization. In paravirtualization, the guest OS is aware that it’s running within a VM. Instead of talking directly to hardware or protected memory, it will talk to the virtualization software. This eliminates the need for full hardware emulation in the virtualization software, greatly improving performance. The downside is that the selection of guest OS is limited to those that support paravirtualization with your virtualization software.

Containers. A container is closer to a chroot’d tree on steroids than a full VM. The software running within the container can only see the files, memory, and processes within the container; however, the kernel is shared among all the containers. Therefore,  all the containers are necessarily running the same OS. Since there is really only one OS running on the whole system, containers have the lowest overhead and best scalability, but they are much more limited in their flexibility.

Virtualization Software

VMware. VMware introduced the first real virtualization solution, and have maintained a significant lead over their competitors since. VMware has a great set of tools to manage VMs, including Lab Manager (managing groups of VMs together), VMmotion (migrating VMs from host to host), and Infrastructure Client (a great view into all the VMs on a set of host servers).

VMware’s biggest downside is cost. Also, they have had several issues with their licensing tools, ranging from the inability to issue a license key for purchased software to updates that caused VMs to not start due to spurious license errors.

The general consensus I’ve heard is that if you can afford VMware, they are the best option for large-scale server virtualization.

Xen. Xen is an open source virtualization solution that most closely competes with VMware. On paper it looks very similar to VMware. In practice the toolset is much less mature, and the product has a lot of rough edges.

KVM. KVM is virtualization software implemented as a Linux kernel module. Because it is fully Linux, many Linux distributations have announced that KVM will be their preferred virtualization solution going forward. Today, it is still a work in progress, and not yet ready for datacenter deployment. KVM supports

OpenVZ. OpenVZ is a container-only solution. If your virtualization needs can be satisfied by containers, OpenVZ is worth considering. For most virtualization needs, though, OpenVZ is not enough.

Hyper-V. Hyper-V is Microsoft’s server virtualization solution. I don’t know much about it, and it was (unsurprisingly) not talked about much at LinuxWorld.

The conclusion to the mystery of the haunted inn.

As expected, the ghosts rumors were started by the businessman who wanted to buy the inn, and he had someone on the inside helping him out. What I didn’t expect was that the businessman was trying to cover up a murder from 15 years ago, and that at least some of the ghosts were real.

It was nice to see all four “Scooby Gang” members helping out. While Rui was the first to put all the pieces together, Ran, Midori, and Rin all contributed vital clues. It was also nice to see Ran and Midori use their powers for more than just telepathy. In this episode, their powers were essential to figuring out the mystery in time.

I hope the episodes get a little less Scooby-Doo-like — if they do the mystery of the abandoned theme park or the monster in the lake I may have to stop watching.  I watched Scooby Doo as a kid; I don’t feel like rewatching it in anime form.

I attended LinuxWorld this year as a full conference attendee, thanks to a professional growth and development opportunity from my employer. Caroline and I took advantage of the fact that the Moscone Convention Center in San Francisco was within half a mile of the San Francisco WorldMark resort, and drove up there the weekend before to do some site seeing. During LinuxWorld Caroline and Mai met me for lunch, and then after the day’s sessions were over we did more site seeing and found interesting restaurants for dinner.

I had three goals for this trip. First, have fun with Caroline and Mai in San Francisco. Second, pick up new technical skills and new ideas at LinuxWorld. Third, take lots of great photos around San Francisco, with a enough top-notch photos to put together a 12-month San Francisco-themed calendar. I think I succeeded at all three.

Caroline, Mai, and I walked around many beautiful places in San Francisco, ate at some delicious restaurants, and generally had a great time in the city. I’ll leave the specific restaurant reviews to Caroline (http://www.yelp.com/user_details?userid=c7j-0cX26jvSu5oa7HwnWw), but overall we loved almost every place we went to. We walked around the San Francisco side of the Golden Gate Bridge / Fort Point, Union Square, Chinatown, Japan Town, much of the waterfront, the Ferry Building, and much more. (Incidentally, I think I need to replace my sneakers after that much hiking in them.) Mai was very well behaved almost all the time, and we didn’t have any significant problems in any of the restaurants. She did get cold a couple of times — the temperature in SF in August hovers in the 50s and 60s, with a chill wind and fog. Caroline even admitted she had a good time.

LinuxWorld had a lot of interesting sessions. I think I chose a good mix of specific technical sessions (DRBD, Heartbeat, etc) and conceptual (virtualization options, system management, etc). Virtualization was definitely a hot topic, for good reason; while it adds a lot of complexity, on balance it can make IT’s job much easier if implemented well. (Of course, it can turn into a giant nightmare if it’s implemented without sufficient planning.) With RedHat and SUSE having separate conferences, a lot of the focus was on Ubuntu. Personally, I prefer RHEL/CentOS as my server OS, but it’s not like there’s that much difference at core between the distributions.

I won an iPod Touch 8GB from Rackspace for placing first place in their sysadmin challenge. My time was 8 minutes, which beat the #2 person by a respectable margin. (I think #2 was 12 or 14 minutes). The challenge seemed simple enough — Rackspace provided you a Linux VM and gave you a list of 4 simple tasks that you had to do in under 20 minutes. Of course, problems on the system prevented you from doing those tasks, so you had to fix the problems in order to complete the tasks. It was a pretty good breadth-of-knowledge test — each problem was simple if you’ve seen it before, and somewhat challenging to near impossible if you had no experience in that area of Linux sysadmin. I’ll have to remember this technique if I’m ever asked to interview sysadmins in the future.

The photography went well too. Right before the convention Palm unexpectedly sent me a bonus check for a patent application filing, which I used to buy the Sigma 50-150/2.8 lens I’ve been eying. In addition to that, I also brought my Sigma 18-50/2.8 and my Sigma 30/1.4. The 50-150 made a great walkabout lens on Saturday and Sunday; it helped me get some great birds-in-flight photos. The 30/1.4 was perfect walking around at night and in restaurants — I was able to get some great photos handheld in low light that wouldn’t have been possible without a tripod. Caroline’s patience with my photography bordered on saint-like, which gave me to the time to get most of the shots I wanted.

On the last night, at about 9pm, I filled my 16GB flash card (talk about good timing!), and ended up with about 1300 photos shot. We’ve started going through them, but it will be a little while till they get posted. I expect about 100-130 photos good enough to post online, and I’m pretty confident at least 12 will be good enough for a calendar.

Nogizaka Haruka no Himitsu likes to take stock anime tropes and make them interesting and different. For example, Yuuto lives with his big sister and a sexy teacher, which is not atypical for an anime. However, unlike most shows where the guy is pampered, here the guy is the one who cooks, cleans, and takes care of the lazy slob girls. Also, the sexy teacher makes advances (typical anime), but they are pathetic and and he’s not the slightest bit interested (unusual for an anime).

In this episode, the shows takes the stock “shadow someone who’s acting wierd to see what they’re up to” plot and makes it fresh and fun. It was amusing to watch the group’s antics, from Haruka getting distracted by the mirror to Mika and Nanami forgetting completely about the task at hand and going shopping. Haruka’s innocence (“what’s a threesome?”) was entertaining. The self-referential humor (“urusai, urusai, urusai! I always wanted to say that” and the whole “It’s Ueda Kana!” bit) had me laughing out loud. How often does a show have a voice actress play a character that needs to imitate the voice actress’ real voice, for logical, in-plot reasons?

Of course, the whole thing was a misunderstanding, but in the process we learned more about Hazuki’s relationship to the Nogizaka family, have Yuuto (literally) run into Shiina again, and of course watch Haruka and Yuuto’s relationship progress. Somehow, though, I don’t think this is how Yuuto wanted Haruka to meet his sister.