Over the past few years in graduate school, I've somehow ended up managing the computing resources for our lab. Prior to joining the lab, I didn't have any real system administration experience, so while I've learned a lot over these years, I'm still learning plenty whenever something breaks or we decided to upgrade our resources. As you might expect, there's a huge amount of complexity in managing a linux cluster, and one often has to just trust various suggestions at face value without fully understanding the underlying logic. Regardless, I always try to make a point of having at least some idea of why various best practices have been established. The following is a story about my not doing that, and the minor annoyance that it's causing.
About a year ago, we purchased 32 compute nodes for a new cluster. Even if you've never been in a server room before, you've probably seen one on TV (and in the picture included in this post). Basically, there are a bunch of metal racks lined up next to each other, and the computers are stacked on top of each other inside the racks. The racks in our lab can hold about 40 computers each. Cooled air is blown up from the floor in front of the computers, then pulled over the processors by fans inside the computers.
Anyway, when we bought these computers, the company from whom we purchased them came out to do the installation. As I said, we had purchased 32 machines and had room for 40 machines in the rack, so we were going to have 8 empty slots in the rack. I was a little surprised to see that the team installed the computers with all 8 empty spots placed at the top of the rack. When I asked about this, they told me that installing machines from the floor up is "The Rocks Way" (
Rocks is a version of linux designed for high performance computing clusters). When I prodded a little further , they told me that it was done to improve the airflow through the computers: put all the machines as close to the air conditioning as possible, and create a continuous block of machines to get even airflow. Satisfied with their answer, I left them to their work.
Somewhat later I noticed that by stacking the machines from the floor up, the computer named node-01 was placed on the bottom with node-32 on top. Previously in the lab, we'd always installed our machines in the opposite order (01 on top). For consistency's sake, I asked if there was a good reason to have 01 on the bottom. Again, I was told that this was "The Rocks Way," but when I prodded further, I didn't really get any further explanation. As such, we asked them to rename the machines to stay in line with the rest of our layout.
And now we get to the point of this unnecessarily long and boring story. Our lab is about to purchase 8 more computers. We want to add them to this cluster. They will be numbered 33-40. But if we put them into our rack, our nicely number machines are suddenly going to be out of order. From top to bottom, they will read 33-40,1-32. Obviously, that's not a huge deal for those who aren't as anal retentive as myself, but if we'd just done things the Rocks way (and understood why), we would now have nice sequentially numbered machines.
It's pretty safe to say that almost every best practice has some reasoning behind it. In plenty of cases, this reasoning might not apply to your situation; but if you don't find out what the reason was in the first place, you've got no hope of determining whether it is relevant to you.
