{"id":475,"date":"2010-08-03T20:01:51","date_gmt":"2010-08-03T19:01:51","guid":{"rendered":"http:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/"},"modified":"2010-08-03T20:01:51","modified_gmt":"2010-08-03T19:01:51","slug":"puppet-custom-facts-and-master-less-puppet-deployment","status":"publish","type":"post","link":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/","title":{"rendered":"Puppet custom facts, and master-less puppet deployment"},"content":{"rendered":"

As I mentioned a few weeks ago, I'm using Puppet<\/a> for some smaller projects here at work. They're pilot projects to see how puppet behaves and scales for us before taking it into bigger challenges.<\/p>\n

One of the problems so far is that we're using fabric<\/a> as the "last-mile" deployment tool, and that doesn't yet have any way to run jobs in parallel. That's the reason why I'm starting to look elsewhere, like mcollective<\/a> for example.<\/p>\n

However, today I had to prepare a new varnish<\/b> box for files.myopera.com. This new machine is in a different data center than our current one, so we don't have any puppetmaster deployed there yet. This stopped me from using puppet in also another project<\/a>. But lately I've been reading on the puppet-users mailing list that several people have tried to deploy a master-less puppet configuration, where you have no puppetmasterd running<\/b>. You just deploy the puppet files, via rsync, source control or pigeons, and then let the standalone puppet executable run<\/b>.<\/p>\n

Puppet master-less setup<\/h3>\n

To do this, you have to at least have a good set of reusable puppet modules, which I tried to build small pieces at a time during the last few months. So I decided to give it a shot, and got everything up and running quickly. Deployed my set of modules in \/etc\/puppet\/modules<\/code>, and built a single manifest file that looks like the following:<\/p>\n

\r\n#\r\n# Puppet standalone no-master deployment\r\n# for files.myopera.com varnish nodes\r\n#\r\nnode varnish_box {\r\n\r\n    # Basic debian stuff\r\n    $disableservices = [ "exim4", "nfs-common", "portmap" ]\r\n    service { $disableservices:\r\n        enable => "false",\r\n        ensure => "stopped",\r\n    }\r\n\r\n    # Can cause overload on the filesystem through cronjobs\r\n    package { "locate": ensure => "absent", }\r\n    package { "man-db": ensure => "absent", }\r\n\r\n    # Basic configuration, depends on data center too\r\n    include opera\r\n    include opera::sudoers\r\n    include opera::admins::core_services\r\n    include opera::datacenters::dc2\r\n\r\n    # Basic packages now. These are all in-house modules\r\n    include base_packages\r\n    include locales\r\n    include bash\r\n    include munin\r\n    include cron\r\n    include puppet\r\n    include varnish\r\n\r\n    varnish::config { "files-varnish-config":\r\n        vcl_conf => "files.vcl",\r\n        storage_type => "malloc",\r\n        storage_size => "20G",\r\n        listen_port => 80,\r\n        ttl => 864000,\r\n        thread_pools => 8,\r\n        thread_min => 800,\r\n        thread_max => 10000,\r\n    }\r\n\r\n    #\r\n    # Nginx (SSL certs required)\r\n    #\r\n    include nginx\r\n\r\n    nginx::config { "\/etc\/nginx\/nginx.conf":\r\n        worker_processes => 16,\r\n        worker_connections => 16384,\r\n        keepalive_timeout => 5,\r\n    }\r\n\r\n    nginx::vhost { "https.files.myopera.com":\r\n        ensure => "present",\r\n        source => "\/usr\/local\/src\/myopera\/config\/nginx\/sites-available\/https.files.myopera.com",\r\n    }\r\n\r\n    bash:: prompt { "\/root\/.bashrc":\r\n        description => "Files::Varnish",\r\n        color => "red",\r\n    }\r\n\r\n    munin:: plugin::custom { "cpuopera": }\r\n\r\n    munin:: plugin { "if_eth0":\r\n        plugin_name => "if_"\r\n    }\r\n\r\n    munin:: plugin {\r\n        [ "mem_", "load", "df", "df_inode", "netstat", "vmstat",\r\n          "iostat", "uptime", "threads", "open_files", "memory", "diskstats" ]:\r\n    }\r\n}\r\n\r\nnode default inherits varnish_box {\r\n}\r\n\r\nnode 'my.hostname.opera.com' inherits varnish_box {\r\n}\r\n\r\n<\/code><\/pre>\n
This manifest installs varnish, nginx, a bunch of basic packages I always want on every machines (vim, tcpdump, etc…), munin and appropriate plugins already configured, and also a nice red bash prompt<\/span> to warn me that this is production stuff.<\/p>\n
This file is everything the puppet client needs to run and produce the desired effect, without needing a puppet master. Save it as varnish-node.pp<\/code> and then you run it with:<\/p>\n
\r\npuppet varnish-node.pp\r\n<\/code><\/pre>\n
One problem that usually arises is how to serve the static files. In this case, I assumed I'm going to check out the source code and config files from my own repository into \/usr\/local\/src\/...<\/code> so I don't need to point puppet to a server with the classic:<\/p>\n
\r\nsource => "puppet:\/\/\/module\/filename"\r\n<\/code><\/pre>\n
but you can just use:<\/p>\n
\r\nsource => "\/usr\/local\/whatever\/in\/my\/local\/filesystem"\r\n<\/code><\/pre>\n
That's great and it works just fine.<\/p>\n
Custom facts<\/h3>\n
Puppet uses a utility called facter<\/b> to extract "facts" from the underlying system, sysinfo-style. A typical facter run produces the following output:<\/p>\n
\r\n$ facter\r\narchitecture => x86_64\r\ndomain => internal.opera.com\r\nfacterversion => 1.5.6\r\nfqdn => cd01.internal.opera.com\r\n...\r\nhardwaremodel => x86_64\r\nhostname => cd01\r\nid => cosimo\r\nipaddress => 10.20.30.40\r\nipaddress_eth0 => 10.20.30.40\r\nis_virtual => false\r\n...\r\nkernel => Linux\r\nkernelmajversion => 2.6\r\n...\r\noperatingsystem => Ubuntu\r\noperatingsystemrelease => 10.04\r\nphysicalprocessorcount => 1\r\nprocessor0 => Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz\r\nprocessor1 => Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz\r\nprocessorcount => 2\r\n...\r\n<\/code><\/pre>\n
and so on. Within puppet manifests, you can use any of these facts<\/b> to influence the configuration of your system. For example, if memorysize > 4.0 Gb<\/code> then run varnish with 2000 threads instead of 1000. This is all very cool, but sometimes you need something that facter doesn't give you by default.<\/p>\n
That's why facter can be extended.<\/p>\n
I tried creating a datacenter.rb<\/code> facter plugin that would look at the box IP address and figure out in which data center we're located. That in turn can be used to setup the nameservers and other stuff.<\/p>\n
Here's the code. My Ruby-fu is less than awesome:<\/p>\n
\r\n#\r\n# Provide an additional 'datacenter' fact\r\n# to use in generic modules to provide datacenter\r\n# specific settings, such as resolv.conf\r\n#\r\n# Cosimo, 03\/Aug\/2010\r\n#\r\n\r\nFacter.add("datacenter") do\r\n    setcode do\r\n\r\n        datacenter = "unknown"\r\n\r\n        # Get current ip address from Facter's own database\r\n        ipaddr = Facter.value(:ipaddress)\r\n\r\n        # Data center on Mars\r\n        if ipaddr.match("^88.88.88.")\r\n            datacenter = "mars"\r\n\r\n        # This one on Mercury\r\n        elsif ipaddr.match("^99.99.99.")\r\n            datacenter = "mercury"\r\n\r\n        # And on Jupiter\r\n        elsif ipaddr.match("^77.77.77.")\r\n            datacenter = "jupiter"\r\n        end\r\n\r\n        datacenter\r\n    end\r\nend\r\n<\/code><\/pre>\n
However, there's one problem. When puppet runs, it doesn't get the new fact, even though facter from the command line can see it and execute it just fine (when the plugin is in the current directory).<\/p>\n
Now I need to know how to inform puppet (and\/or facter) that it has to look into one of my puppet modules' plugin<\/code> (or lib<\/code> from 0.25.x) directory to load my additional datacenter.rb<\/code> fact.<\/p>\n
Any idea ??<\/p><\/p>\n","protected":false},"excerpt":{"rendered":"
As I mentioned a few weeks ago, I'm using Puppet for some smaller projects here at work. They're pilot projects to see how puppet behaves and scales for us before taking it into bigger challenges. One of the problems so far is that we're using fabric as the "last-mile" deployment tool, and that doesn't yet […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[388,231,347,390,260,389,76,60],"yoast_head":"\nPuppet custom facts, and master-less puppet deployment - Random hacking<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Puppet custom facts, and master-less puppet deployment - Random hacking\" \/>\n<meta property=\"og:description\" content=\"As I mentioned a few weeks ago, I'm using Puppet for some smaller projects here at work. They're pilot projects to see how puppet behaves and scales for us before taking it into bigger challenges. One of the problems so far is that we're using fabric as the "last-mile" deployment tool, and that doesn't yet […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\" \/>\n<meta property=\"og:site_name\" content=\"Random hacking\" \/>\n<meta property=\"article:published_time\" content=\"2010-08-03T19:01:51+00:00\" \/>\n<meta name=\"author\" content=\"cosimo\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cosimo\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\"},\"author\":{\"name\":\"cosimo\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"headline\":\"Puppet custom facts, and master-less puppet deployment\",\"datePublished\":\"2010-08-03T19:01:51+00:00\",\"dateModified\":\"2010-08-03T19:01:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\"},\"wordCount\":582,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"keywords\":[\"datacenter\",\"deployment\",\"facter\",\"master-less\",\"puppet\",\"puppetmasterd\",\"servers\",\"varnish\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\",\"name\":\"Puppet custom facts, and master-less puppet deployment - Random hacking\",\"isPartOf\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\"},\"datePublished\":\"2010-08-03T19:01:51+00:00\",\"dateModified\":\"2010-08-03T19:01:51+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Puppet custom facts, and master-less puppet deployment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#website\",\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/\",\"name\":\"Random hacking\",\"description\":\"Assume nothing. Code defensively. Keep it simple, stupid!\",\"publisher\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1\",\"name\":\"cosimo\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g\",\"caption\":\"cosimo\"},\"logo\":{\"@id\":\"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Puppet custom facts, and master-less puppet deployment - Random hacking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/","og_locale":"en_US","og_type":"article","og_title":"Puppet custom facts, and master-less puppet deployment - Random hacking","og_description":"As I mentioned a few weeks ago, I'm using Puppet for some smaller projects here at work. They're pilot projects to see how puppet behaves and scales for us before taking it into bigger challenges. One of the problems so far is that we're using fabric as the "last-mile" deployment tool, and that doesn't yet […]","og_url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/","og_site_name":"Random hacking","article_published_time":"2010-08-03T19:01:51+00:00","author":"cosimo","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cosimo","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#article","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/"},"author":{"name":"cosimo","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"headline":"Puppet custom facts, and master-less puppet deployment","datePublished":"2010-08-03T19:01:51+00:00","dateModified":"2010-08-03T19:01:51+00:00","mainEntityOfPage":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/"},"wordCount":582,"commentCount":0,"publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"keywords":["datacenter","deployment","facter","master-less","puppet","puppetmasterd","servers","varnish"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/","url":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/","name":"Puppet custom facts, and master-less puppet deployment - Random hacking","isPartOf":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website"},"datePublished":"2010-08-03T19:01:51+00:00","dateModified":"2010-08-03T19:01:51+00:00","breadcrumb":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/2010\/08\/puppet-custom-facts-and-master-less-puppet-deployment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.streppone.it\/cosimo\/blog\/"},{"@type":"ListItem","position":2,"name":"Puppet custom facts, and master-less puppet deployment"}]},{"@type":"WebSite","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#website","url":"https:\/\/www.streppone.it\/cosimo\/blog\/","name":"Random hacking","description":"Assume nothing. Code defensively. Keep it simple, stupid!","publisher":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.streppone.it\/cosimo\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/c443bedbf6ecf99550d6395620801df1","name":"cosimo","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb1d938720df45a2720724aae99e3bfc?s=96&r=g","caption":"cosimo"},"logo":{"@id":"https:\/\/www.streppone.it\/cosimo\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.streppone.it\/cosimo\/blog\/author\/cosimo\/"}]}},"_links":{"self":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/475"}],"collection":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/comments?post=475"}],"version-history":[{"count":0,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/posts\/475\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/media?parent=475"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/categories?post=475"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.streppone.it\/cosimo\/blog\/wp-json\/wp\/v2\/tags?post=475"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}