Edward Capriolo

Saturday Jul 17, 2010

Using puppet to randomly splay cassanda compaction times

Cassandra stores data in a series of sstables. sstables are write once files. When the sizes of files reach certain thresholds cassandra compacts multiple sstables into one larger sstable.

Since sstables are never modified cassandra does deletes by using tomstones. A tombstone is a marker that says something to the effect of 'data has been deleted'.

The only process that truly removes the data and the sstables from cassandra is called MAJOR COMPACTION. You can manually trigger this compaction with nodetool

/usr/local/cassandra/bin/nodetool -h localhost -p 8585 compact

So far so good, but the devil is in the details. If you have a cluster of 20 nodes how to you run compaction for all of them, but ideally you want to offset or splay the times compaction runs at so each runs at different times.

While you could manage 20 different crontabs with the puppet fileserver that seems kinda of like a waist of time and effort.

Puppet does have cron object...

  cron { "compact":
    command => "/usr/local/cassandra/bin/nodetool -h ring -p 8585 compact",
    user => root,
    hour => 4,
    minute => 11,
    weekday => 3
  }

still this causes all the compactions to run at once.

enter fqdn_rand

This generates 'random' numbers based on the fqdn

$aminute = fqdn_rand(60)
$ahour = fqdn_rand(24)
$awday = fqdn_rand(7)

  cron { "compact":
    command => "/usr/local/cassandra/bin/nodetool -h localhost -p 8585 compact",
    user => root,
    hour => $ahour,
    minute => $aminute,
    weekday => $awday
  }

Results

[root@cdbsd02 ~]# crontab -l
# HEADER: This file was autogenerated at Sat Jul 17 12:56:05 -0400 2010 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: compact
56 21 * * 0 /usr/local/cassandra/bin/nodetool -h localhost -p 8585 compact

 [root@cdbsd04 ~]# crontab -l
# HEADER: This file was autogenerated at Sat Jul 17 13:32:05 -0400 2010 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: compact
55 23 * * 1 /usr/local/cassandra/bin/nodetool -h localhost -p 8585 compact

As you can see all the crontabs run at different times.



Friday Jul 16, 2010

Func & Hadoop the end of start-all.sh


If you follow the stock hadoop docs for setting up, they suggest constructing and ssh keys from your name node or management node to your other nodes so that they can startup datanode and task tracker components. Hadoop itself does not use SSH keys other then for this startup.  This article will show another tactic.

For reference the way to locally start and stop a hadoop component and daemon is:

cd /home/edward/hadoop/hadoop-0.20.2
bin/hadoop-daemon.sh start namenode

If you want to run in foreground:

cd /home/edward/hadoop/hadoop-0.20.2
bin/hadoop namenode

Note: "Edward Capriolo hates using SSH KEYS for software interaction" 

Why do i dislike ssh-keys? I have heard and seen "horror stories" of chains of computers linked together by ssh-key mechanism, where one computer auto-logs into the next to move a file, or run a remote command. Come on get real! if you have a process like this, make your own protocol, use a message queue , or rpc.

hadoop does not use ssh keys in any way other then for the start-all scripts. These scripts leave lots to be desired, mostly they do not provide a basic way to start or stop one node, and they don't have a switch 'status all' or only call start if not already running.

Enter func  https://fedorahosted.org/func/

In a nutshell, func is "for ( computer : list of computer) { run command on computer } "

Func uses a client server certificate management with a CA. So it is more advanced then say just throwing pubkey signatures in /root/.ssh/authorized_keys. Func does not have to run as root either

As I mention all the time I have a private RPM repo. It is uber awesome. Find these rpms and throw them in there.

certmaster-0.24-1.el5.noarch.rpm
func-0.24-1.el5.noarch.rpm
pyOpenSSL-0.6-1.p24.7.2.2.x86_64.rpm
python-simplejson-2.0.5-1.el5.rf.x86_64.rpm

Follow the func docs to get the master/cert server setup. If you are clever you can have func and puppet share the same CA (I am not so I did not)

chkconfig certmaster on

Now we go about installing funcd everywhere we want it. Well great news we already have puppet so this is a breeze

vi /etc/puppet/manifests/func.pp

class funcserver{
  package { ["pyOpenSSL","python-simplejson","certmaster","func"]: ensure => installed }
  file { "/etc/certmaster/minion.conf" :
    owner => root,
    group => root,
    path => "/etc/certmaster/minion.conf",
    source => "puppet:///mainfiles/func/minion.conf" ,
    require => Package[certmaster]
  }
  service {
    funcd:
      ensure => true,
      enable => true,
      subscribe => [ File["/etc/certmaster/minion.conf"], Package[func] ] ,
      require =>  File["/etc/certmaster/minion.conf"]
  }
}

As you can see here we are installing the func packges, and using puppet to start the service and ensure that it is always running. The only customization we need here is we have to push our minion conf onto clients ( this tells them where the func server is)

[main]
certmaster = funcserver.jointhegrid.com
log_level = DEBUG
cert_dir = /etc/pki/certmaster

Push func out with puppet

node 'had01.hadoop.pvt','had02.hadoop.pvt','had03.hadoop.pvt','had04.hadoop.pvt','had05.hadoop.pvt',
     'had06.hadoop.pvt', 'had07.hadoop.pvt' ,'had08.hadoop.pvt', 'had09.hadoop.pvt', 'had10.hadoop.pvt',
     'had11.hadoop.pvt', 'had12.hadoop.pvt', 'had13.hadoop.pvt', 'had14.hadoop.pvt', 'had15.hadoop.pvt',
     'had16.hadoop.pvt', 'had17.hadoop.pvt', 'had18.hadoop.pvt', 'had19.hadoop.pvt', 'had20.hadoop.pvt'

{
  include hadoop_tasktracker_cleanup,standardserver,hadoop_prod_conf
  include funcserver
}

Lets test to see what puppet is going to do notice no-op

[root@had01 ~]# puppetd --server xxxxxxx--waitforcert 60 --test --noop
info: Caching catalog for had01.hadoop.pvt
info: Applying configuration version '1279293057'
notice: //funcserver/Package[func]/ensure: is absent, should be present (noop)
info: //funcserver/Package[func]: Scheduling refresh of Service[funcd]
notice: //funcserver/Package[certmaster]/ensure: is absent, should be present (noop)
notice: //funcserver/File[/etc/certmaster/minion.conf]/ensure: is absent, should be file (noop)
info: //funcserver/File[/etc/certmaster/minion.conf]: Scheduling refresh of Service[funcd]
notice: //funcserver/Service[funcd]/ensure: is stopped, should be running (noop)
notice: //funcserver/Service[funcd]: Would have triggered refresh from 2 dependencies
notice: Finished catalog run in 5.53 seconds

Looks good to me!

puppetd --server xxxxxxx--waitforcert 60 --test --noop

Take a lunch break

Seriously, you deserve it your soo f'in smart. Soon you are going to be able to run commands across your network without ever leaving your machine. Go get yourself a $10 coffee and toast your brilliance.

Signing clients

So after your coffee, all your clients should have func installed on them. You now need to sign the certs.

Signing certs

# certmaster-ca --list
had01.hadoop.pvt

#certmaster-ca --sign had01.hadoop.pvt
/var/lib/certmaster/certmaster/csrs/had01.hadoop.pvt.csr signed - cert located at /var/lib/certmaster/certmaster/certs/had01.hadoop.pvt.cert

Using func
 

# func had*.hadoop.pvt call command run "df -h /"
('had01.hadoop.pvt',
 [0,
  'Filesystem            Size  Used Avail Use% Mounted on\n/dev/sda2              95G   12G   79G  13% /\n',
  ''])
('had02.hadoop.pvt',
 [0,
  'Filesystem            Size  Used Avail Use% Mounted on\n/dev/sda2              95G  8.2G   82G  10% /\n',
  ''])

Conclusion

So now we can manage our systems remotely and specify targets with * expressions!

Hints

Do not run "rm -rf /"

Calendar

Feeds

Search

Links

Navigation