If you have a passion to work at a fast moving company, understand the unique demand of supporting a fully automated scalable backend then you are the person for this job!

Go ahead and submit your resume to me zuhaib.siddique oxygencloud.com . You can read the full description of the job here: http://www.oxygencloud.com/careers

A Systems Administrator position here at Oxygen encompasses much, much more than just maintaining some Linux servers. You will be given the the opportunity to work with the latest technologies and tools in order to overcome the real-world challenges that accompany the ambitious goal of bringing Oxygen to the masses. We are looking for very smart, very dedicated individuals that like nothing more than to get their hands dirty addressing scaling, performance, and availability challenges across a vast cloud infrastructure. Passion, skill, and a whole lot of Geek Cred are the attributes we are looking for here. Does 90% of your home electric bill come from powering dozens of gadgets and several Linux boxes? Does your idea of a “fun night” involve opening your wireless access up to your bandwidth stealing neighbors, so that you can invert all of their web content? Do you find that your most common thought on a daily basis is “I’ll just write a script for that”? If so, then we want to talk to you. Role/Responsibilities: Manage the day-to-day operations of Oxygen’s infrastructure to ensure smooth 24×7 operation Work closely with Engineering to ensure proper and timely deployment of software to staging and production environments. Use cutting edge methods and technology to identify and solve problems related to performance, scalability, reliability, and availability Develop, implement, and continually improve upon best practice processes and procedures Skills/Experience: 2-4 years of hard core system administration or technical operations experience Very strong knowledge of Linux system administration practices and troubleshooting. Strong passion for Cloud storage and Cloud computing technologies Experience with Apache, Tomcat, MySQL, and cloud services infrastructures Ability to troubleshoot errors and proactively identify issues Demonstrated ability to generate and maintain technical documentation Bonus Skills: Understanding of MySQL performance characteristics and scaling options Knowledge of systems architecture design and implementation Knowledge of configuration management and automation (Chef, Puppet, etc.) Experience with the following tech: Nginx, Ruby, Python, LDAP/AD, DNS/BIND, SSO, SMTP, NFS, Beanstalk, Memcached, SSL Requirements for applying: Tell us what interests you about this position Tell us about a project that you have significantly contributed to and are very proud of E-mail the above and your resume in PDF format to jobs@oxygencloud.com

mysqldump -P <port#> -h <server ip>-u mysql_user -p database_name table_name > backup.sql

To restore your DB in the cloud you use a more well-known mysql technique

mysql -u mysql_user -p -h <server ip> cloud_db_name < backup.sql

diff <(ssh server1 'sudo ls -1aR /var/www/vhosts/domain.com') <(ssh server2 'sudo ls -1aR /var/www/vhosts/domain.com')

Update: So a lot of people in the ServerFault page I linked to mentioned rync with the –dry-run flag. I did not look in to this much but just as I posted this blog the first @reply i got on twitter was from @EvanHoffman asking if I had looked at using rsync. Ok so I thought I should revisited the problem and using rsync and I got the follow:

rsync --dry-run -rvce "ssh -p port#" user@server1:/var/www/vhosts/ /var/www/vhosts/domain.com/

Why VMWare ESXi

VMWare is a big player in the VM World so I thought this would be a good place to start.  Also the fact its a Hypervisor that supports Windows, something XEN and KVM cant do.  And the reason it beat out Virtualbox is because it needs a Host OS.  I did not feel like managing another OS to keep my VMs runnings but now that I am done I might go back and do it as a Virtualbox built on top of CentOS or Ubuntu.  Another big plus of VM over a physical NAS is the fact I get a console for my VM’s.  On servers you get things like DRAC from Dell which gives you a console over network but I dont have that luxury.  This give me the ability to work on a server without having to drag a monitor. But nothing comes without downsides, and the larget for ESXi is the fact many of its tools are Windows only.  The vCenter is Windows only meaning me on OS X I have to use a Windows VM to manage my ESXi server.  The vConverter is Linux but then you get another downside in that it seems its does not work for ESXi 4.1.  Which is why I started this project on ESXi 4.0 because that is the lastest version that will work for P2V.  This is why I think if i was to do it again from the ground up I will do it with Virtualbox and start with a new system.  I ended up ditching the P2V after I nuked the drive (doh!).

Hooking up Physical Drive

One thing a NAS needs is a bunch of drives.  The best setup is a RAID but my box is ghetto rig of pretty much old parts and ESXi did not support the built in RAID so my system has two large drives.  But part of the problem is I have data on the drives as it was being used as a physical system and I did not want them to be locked in to ESX and wanted them to connect to the VM as physical drives. This turned out to be much harder then I expected and I went as far as asking the question on ServerFault and even there it could not get me the right answer.  I ended up finding the answer in the arstechnica forums and was able to hook up my two physical drives to my VM.

Projects

Get a Nagios VM setup to monitor the NAS, that is just for the fun because if the box goes down I dont know what happen Exchange 2010 Trial

Problems/Bugs

Still my Ubuntu VM will sometimes crash without any error and nothing on console, was hoping this switch would solve that or let me find the problem.  Also in the three weeks my system has been up I have had one issue where ESXi itself crashed which is a PITA.

I am currently helping LinkedIn move its users from Exchange 2003 to Exchange 2010, and if anyone is thinking why Exchange I asked the same questions my first day on the contract and after hearing the answers its a bit clear. Zimbra has some upside but its still ruff around the edges, and in a environment that has both developers who can live with that, but regular employees who have spent years learning the Outlook world its hard to make the changes.

While most of LinkedIn users are Macs, a few are testing Linux for development boxes. And the question that came up, is their a Linux E-Mail client that is currently supporting Exchange 2007+ as Microsoft has changed the way it does Calendaring. While we can get away with IMAP for all the E-Mail issue, Calendaring is important for a company as that is the way you book rooms, make sure people are free and etc. I posed the question on Quora, invite only, http://www.quora.com/Is-there-any-Linux-email-client-that-will-work-with-Exchange-2010 and got nothing. The best was someone saying Evolution but its support ended with 2003, which is what we are moving away from. Googling did not help much, usually you found most people asking the same questions and the answer was mostly use OWA, Exchanges new Web Mail which is very good and now supports Firefox and Safari with AJAX-ish style.

But some people want a native client. Then Brian Guan, an Engineer at Linkedin and Linux user found xdata-provider which is a plug in for Thunderbird Lightning to add Exchange Calendar support, and since its a plug in it means you can also use it on any OS that runs Thunderbird, not just Linux!. After some tweaking and configuring we got it to work prefect, and pretty much all features worked. You could check availability of other people, book rooms and everything. It crashed a few times and stopped work once or twice, but checking in with Brian today it still works. And from what I see on basic google search we might be the first company to have actually done a Exchange 2010 + Linux setup, as xdata does not provide much info on their own deployment. I hope to take a look of the code of xdata and keep it update, as well as it might help the adoption of Linux in the corporate world as people will have at lest now a working Calendar solution for Linux.

Some background on iFolder and how did i find it, a client came up to me with an unique issue. They have an ongoing legal matter that requires them to share large amounts of pdf’s and scanned documents with their lawyers. I run their private E-Mail server for them so I have them setup with high limits for incoming and outgoing attachments, but most other system admins dont because they dont want to clog up their networks with people sending E-Mails of their cats. Also most 3rd party ISPs and e-mail providers (like gmail) put hard caps between 10mb to 30mb which confuses more things. So they needed a way to put files up on a server, shared with others, be secure, and most importantly be dead simple. In comes iFolder! Now like I said a service like iFolder is Dropbox, but the problem is its not on your server. This is not a issue to most, but for companies and lawyers it scares the hell out of them. Just listen to TwiL and you get the idea. I am not saying Dropbox is unreliable or its ToS is bad, and I bet they have some strong provisions, i mean I use them for my own data, but for a company that already has invested in a server and connection they feel its better if everything is kept local. There is merit to that, as time and time again we have seen cases were groups will send legal notices to ISPs and its a crap shoot if they forward the request to the client or not, so their is a chance that your data might be handed over because of a warrant without you knowledge because it was hosted on someone else server.

So setting up iFolder is not straight forward as Novell who supports the project right now has built it on top of .NET, which seems an odd choice, meaning you need MONO on Linux/OS X to run the iFolder Server (And Client). Also officially Novell only releases the rpm packages (you can get the source) and support only SuSE, but, you can easily get it working on other Linux distros with some work. We use Ubuntu Server which is great, and a great guide on getting iFolder up and running can be found here http://www.x2b4.com/howto/how-to-install-ifolder-on-ubuntu-server/. Also great reading is iFolder Document which has some great tips on how to get going for an Admin, but thats the problem. They are written for a admin to get going. What about the end user? What are you going to give to help them get going?

The iFolder User Guide is great but a lot of useful info is burred under useless stuff, like LADP settings and etc. Really the user does not care about that, he/she needs to get up and running. So for my client I wrote them a End user guide! I used LaTeX to make the document and included two pdf extract from the user guide to help clear things up for the end user. I also took screen shots to include in the pdf and excerpts from the iFolder User Guide as two PDF files.

You can Download the PDF here: iFolder End User Guide

You can download the LaTeX code, photos, and PDF excerpts from iFolder User Guide here: iFolder End User LaTeX Code