So recently I had the opportunity to present to the ElasticSearch Meetup Group hosted at Atlassian to talk about how HipChat scaled to 1 Billions messages. Below are video and the slides from the talk, hope you enjoy.
As you know I joined Oxygen Cloud a few months back as Senior System Admin and now as we grow we need more in the Operations team as we get ready to launch our new secret project.
If you have a passion to work at a fast moving company, understand the unique demand of supporting a fully automated scalable backend then you are the person for this job!
Go ahead and submit your resume to me zuhaib.siddique
A Systems Administrator position here at Oxygen encompasses much, much more than just maintaining some Linux servers. You will be given the the opportunity to work with the latest technologies and tools in order to overcome the real-world challenges that accompany the ambitious goal of bringing Oxygen to the masses. We are looking for very smart, very dedicated individuals that like nothing more than to get their hands dirty addressing scaling, performance, and availability challenges across a vast cloud infrastructure. Passion, skill, and a whole lot of Geek Cred are the attributes we are looking for here. Does 90% of your home electric bill come from powering dozens of gadgets and several Linux boxes? Does your idea of a “fun night” involve opening your wireless access up to your bandwidth stealing neighbors, so that you can invert all of their web content? Do you find that your most common thought on a daily basis is “I’ll just write a script for that”? If so, then we want to talk to you. Role/Responsibilities: Manage the day-to-day operations of Oxygen’s infrastructure to ensure smooth 24×7 operation Work closely with Engineering to ensure proper and timely deployment of software to staging and production environments. Use cutting edge methods and technology to identify and solve problems related to performance, scalability, reliability, and availability Develop, implement, and continually improve upon best practice processes and procedures Skills/Experience: 2-4 years of hard core system administration or technical operations experience Very strong knowledge of Linux system administration practices and troubleshooting. Strong passion for Cloud storage and Cloud computing technologies Experience with Apache, Tomcat, MySQL, and cloud services infrastructures Ability to troubleshoot errors and proactively identify issues Demonstrated ability to generate and maintain technical documentation Bonus Skills: Understanding of MySQL performance characteristics and scaling options Knowledge of systems architecture design and implementation Knowledge of configuration management and automation (Chef, Puppet, etc.) Experience with the following tech: Nginx, Ruby, Python, LDAP/AD, DNS/BIND, SSO, SMTP, NFS, Beanstalk, Memcached, SSL Requirements for applying: Tell us what interests you about this position Tell us about a project that you have significantly contributed to and are very proud of E-mail the above and your resume in PDF format to firstname.lastname@example.org
As more and more things move to the “cloud” as a Sys Admin you might find your self tasked with managing a Database that you don’t have shell access too. That has been the case for me with my last two jobs, Laughing Squid and now as Senior Sys Admin at Oxygen Cloud. With Oxygen we needed to backup some AWS RDS Instances and wanted to do it using a server on EC2 so we don’t have the bottle next of our local internet. This same method works with Rackspace Cloud Sites which I used at Laughing Squid. Its use a little known (could not find it easily with a google search) trick with mysqldump
mysqldump -P <port#> -h <server ip>-u mysql_user -p database_name table_name > backup.sql
To restore your DB in the cloud you use a more well-known mysql technique
mysql -u mysql_user -p -h <server ip> cloud_db_name < backup.sql
So I wanted to find a quick code that will allow me to a diff over two folders as I want to be sure all files get transfered as we are using Plesk migrator to move domains from one server to another. Found a Serverfault post titled How do diff over ssh? but most of the suggestions are for files and not folders and none that work recursively. Well using this answer from Server Fault and some man pages I have the following
diff <(ssh server1 'sudo ls -1aR /var/www/vhosts/domain.com') <(ssh server2 'sudo ls -1aR /var/www/vhosts/domain.com')
Update: So a lot of people in the ServerFault page I linked to mentioned rync with the –dry-run flag. I did not look in to this much but just as I posted this blog the first @reply i got on twitter was from @EvanHoffman asking if I had looked at using rsync. Ok so I thought I should revisited the problem and using rsync and I got the follow:
rsync --dry-run -rvce "ssh -p port#" user@server1:/var/www/vhosts/ /var/www/vhosts/domain.com/(I have not tested the rsync solution with a live box but from testing on my local system I assume this would work) One of the limitation of this you need to be on one of the two servers vs the solutions using diff is something you do on your computer that can SSH to both systems.
For as long as I have been tinkering with Linux I have always had a server at home for one reason or another. It all started out serving DSLReports.com *nix link list running on Red Hat to now Ubuntu Server that does it all for me on ESXi. But getting to ESXi has not been fun. This blog post is to outline what I had to do to make the switch and maybe help others making the switch.
Why VMWare ESXi
VMWare is a big player in the VM World so I thought this would be a good place to start. Also the fact its a Hypervisor that supports Windows, something XEN and KVM cant do. And the reason it beat out Virtualbox is because it needs a Host OS. I did not feel like managing another OS to keep my VMs runnings but now that I am done I might go back and do it as a Virtualbox built on top of CentOS or Ubuntu. Another big plus of VM over a physical NAS is the fact I get a console for my VM’s. On servers you get things like DRAC from Dell which gives you a console over network but I dont have that luxury. This give me the ability to work on a server without having to drag a monitor. But nothing comes without downsides, and the larget for ESXi is the fact many of its tools are Windows only. The vCenter is Windows only meaning me on OS X I have to use a Windows VM to manage my ESXi server. The vConverter is Linux but then you get another downside in that it seems its does not work for ESXi 4.1. Which is why I started this project on ESXi 4.0 because that is the lastest version that will work for P2V. This is why I think if i was to do it again from the ground up I will do it with Virtualbox and start with a new system. I ended up ditching the P2V after I nuked the drive (doh!).
Hooking up Physical Drive
One thing a NAS needs is a bunch of drives. The best setup is a RAID but my box is ghetto rig of pretty much old parts and ESXi did not support the built in RAID so my system has two large drives. But part of the problem is I have data on the drives as it was being used as a physical system and I did not want them to be locked in to ESX and wanted them to connect to the VM as physical drives. This turned out to be much harder then I expected and I went as far as asking the question on ServerFault and even there it could not get me the right answer. I ended up finding the answer in the arstechnica forums and was able to hook up my two physical drives to my VM.
Get a Nagios VM setup to monitor the NAS, that is just for the fun because if the box goes down I dont know what happen Exchange 2010 Trial
Still my Ubuntu VM will sometimes crash without any error and nothing on console, was hoping this switch would solve that or let me find the problem. Also in the three weeks my system has been up I have had one issue where ESXi itself crashed which is a PITA.
As much as open source guys like myself love to bash on MS, there is just something they do right. And if its not right, sometimes they are just the only guys in town. And one of those things in Microsoft Exchange. No one has yet been able to replicate the full experience of Outlook + Exchange in terms of calendaring and E-Mail, especially if your looking for a non-cloud solution. Hell even many Web 2.0 companies who are ready to accept non-Microsoft products have a hard time walking away from Exchange. But in that case you might run in to a case you might have a Exchange Server but a Client using Linux, and that is what I ran in to helping LinkedIn move to Exchange 2010. So what are you to do? Well we got a fix for you.
If your a cloud type of person, whatever that means, meaning you like to put stuff online and share then you likely have been using Dropbox like I do. Its a free/paid service that lets you store up to 2GB free (and more if you pay) online. Thats not new, but what it also does it sync with local folders, and allows for you to collaborate on those folders and keep tracks of who is editing what. So its great way to share files with friends that you cant normally do with E-Mail. But What if you dont trust keeping your super secret files on some startups server? What if you need something with more control or more size and have sever power to throw at it. Then iFolder is for you! But with all OSS stuff their is no great end user – user guide around (as in the thing for your customer), but read on as I think I have the solution.