You are on page 1of 108

Designing Puppet:

Roles / Profiles Design Pattern

Puppet Camp Stockholm, Feb 2013

Thursday, 7 February 13
Hello
Craig Dunn
Puppet user since 2008 as an IT contractor
Started with 0.24
Joined Puppet Labs in June 2012
@crayfishX
Freenode IRC: crayfishx
Thursday, 7 February 13
Agenda
How people typically design Puppet
Real-world case study
Thinking about components
Designing Puppet for your users
Node classification
Data separation
Thursday, 7 February 13
Background

Originally a blog post written in May 2012


Advocated by many Puppet Labs Engineers
Based on a real world solution
Several community members have adopted
with success

Thursday, 7 February 13
Designing Puppet

You write awesome modules


You classify them to your node

Thursday, 7 February 13
Designing Puppet

Node Classification

Modules

Thursday, 7 February 13
Down the road...
Your infrastructure grows
Business requirements will change
Your Puppet code feels bulky and high
maintenence
There will always be edge cases eventually
You decide it needs refactoring
Thursday, 7 February 13
Danger Signs
Resources being declared in two modules
You dont know where your
implementation fits
Lots of logic at a node level
Repetition and duplication
The if statement is your go-to-guy
Thursday, 7 February 13
Write good modules

Should manage only its own resources


Should be granular
Should be portable

Thursday, 7 February 13
Thinking beyond the
module....

Puppet is a code base


How do I design an effective framework
Gluing everything together

Thursday, 7 February 13
Node-level logic

Risks duplication and repetition


No guarantee of consistency
TMI!

Thursday, 7 February 13
Node-level logic
node basil {
class { apache:
version => latest,
}
class { motd: }
class { ssh: }
class { users:
default_shell => /bin/false,
}
Class[ssh] -> Class[users]
}

Thursday, 7 February 13
Node-level logic

What happens when I have 1000 nodes


Or 10,000 nodes!!
Thats a lot of code!
So where should implement this?

Thursday, 7 February 13
Designing Puppet
Provide business logic to classification
Provide an abstraction layer for
implementation of components
Make code adaptable to complex
requirements
Reduce node-level logic
Reduce functionality overlap
Thursday, 7 February 13
What is the worse
thing that is
going to happen to
your Puppet code?

Thursday, 7 February 13
Business requirements

Thursday, 7 February 13
Business logic does not
align with technology

Thursday, 7 February 13
Case study

Real world problem


Solved through design

Thursday, 7 February 13
Case study
We have 3
applications we
need to deploy using
Puppet

Thursday, 7 February 13
The business view

Application X

Application Y Application Z

Thursday, 7 February 13
Go forth and
Puppetize!

Thursday, 7 February 13
Go forth and
Puppetize!
And we jumped right in...

Thursday, 7 February 13
Things got painful

Thursday, 7 February 13
Problems
These applications arent that different
They seem to share a whole bunch of
similarities
Implementation differed on different
environments and locations
Writing 3 separate modules creates
conflicts and duplication

Thursday, 7 February 13
Our code was hacky

Thursday, 7 February 13
We are trying to code
business logic.

Thursday, 7 February 13
Stop thinking about
what it looks like

Break everything down into components


Granularity is the key
Think about what it actually is

Thursday, 7 February 13
What we realised

Each application stack is a collection of a


subset of the same Java apps implemented
in different ways

Thursday, 7 February 13
The business view

Application X

Application Y Application Z

Thursday, 7 February 13
The technical reality

Application X

ApplicationA
Ypplication Z

Thursday, 7 February 13
We only have one
application

Implemented many
different ways
Thursday, 7 February 13
So we had an idea!

Reduce each Java sub application into


granular Puppet modules
Create a code layer responsible for
implementation
Lets call them profiles

Thursday, 7 February 13
class profiles::x {
include tomcat
include mysql
include componenta
include componentb
componentb::resource { name:
ensure => present,
}
}

class profiles::y {
include tomcat
include mysql
include componenta
include componentc
include componentd
}

class profiles::z {
include tomcat
include mysql
include componenta
include componentb
include componentd
include dependancy
Class[dependancy] -> Class[componentd]
}

Thursday, 7 February 13
Use inheritance for abstraction within profiles
class profiles::application {
include tomcat
include mysql
include componenta
}

class profiles::application::x inherits profiles::application {


include componentb
componentb::resource { name:
ensure => present,
}
}

class profiles::application::y inherits profiles::application {


include componentc
include componentd
}

class profiles::application::z inherits profiles::application {


include componentb
include componentd
include dependancy
Class[dependancy] -> Class[componentd]
}

Thursday, 7 February 13
Profiles and
Components

Resources
Thursday, 7 February 13
Profiles and
Components

Components: Resource modelling

Resources
Thursday, 7 February 13
Profiles and
Components

Profiles : Implementation

Components: Resource modelling

Resources
Thursday, 7 February 13
In reality it was worse

Thursday, 7 February 13
In reality it was worse
2 different deployment types made up of
over 15 server types each

Thursday, 7 February 13
In reality it was worse
2 different deployment types made up of
over 15 server types each
10+ locations

Thursday, 7 February 13
In reality it was worse
2 different deployment types made up of
over 15 server types each
10+ locations
4 environment types

Thursday, 7 February 13
In reality it was worse
2 different deployment types made up of
over 15 server types each
10+ locations
4 environment types
Every installation was an edge case!

Thursday, 7 February 13
In reality it was worse
2 different deployment types made up of
over 15 server types each
10+ locations
4 environment types
Every installation was an edge case!
My slides werent big enough.

Thursday, 7 February 13
Lessons learned

Granularity is good
Dont assume business logic will directly
translate to technology
Abstraction is awesome.... but thats
nothing new....

Thursday, 7 February 13
Abstraction is a core
principle of coding
Functions are abstracted by methods
Methods abstracted by classes and modules
They are abstracted with libraries
Puppet is code!

Thursday, 7 February 13
Puppet is all about
abstraction
Data is abstracted by Hiera
Providers are abstracted by types
Resources are abstracted by classes
Classes are abstracted by modules

Thursday, 7 February 13
Puppet is all about
abstraction
Data is abstracted by Hiera
Providers are abstracted by types
Resources are abstracted by classes
Classes are abstracted by modules
Modules are abstracted by profiles
Thursday, 7 February 13
Focussing on
Abstraction
Weve turned business logic into a
technology stack
Can we translate that back into business
logic?
Why would we even want to do that?

Thursday, 7 February 13
UAT Cluster node
Our example configuration model:

include security
include users
include ntp
include ssh::server
include customapp
include tomcat::server

class { jenkins:
require => Class[tomcat::server],
}

include mysql
database { apptest:
ensure => present,
}

Thursday, 7 February 13
Think about the users
Meet John, Susan and Bill.

Thursday, 7 February 13
John is a Sysadmin

Wants to ensure all servers have kernel


hardening, NTP and SSH Server installed
Wants to manage what packages, services,
files and other resources
Is responsible for maintaining all the
components of a UAT cluster server

Thursday, 7 February 13
Susan is an application
specialist

Cares that a UAT Cluster node requires


MySQL Server, Tomcat Server and Jenkins
server installed.

Thursday, 7 February 13
Bill is an IT manager

Bill cares that the server is a UAT Cluster


node

Thursday, 7 February 13
What do they care
about?

John cares about modelling all resources


Susan cares about the technology stack
Bill cares about the business logic

Thursday, 7 February 13
In Puppet

Resource modelling is done in component


modules
The technology stack is defined in profiles
Where do we represent the business logic
for Bill?

Thursday, 7 February 13
Introducing Roles

Represent business logic, not technology


Define a set of technology stacks (profiles)
that make up the logical role
Allow the business to manage how the
infrastructure looks without defining what it
is

Thursday, 7 February 13
A node can only have
one role
A role can include as many profiles as
required to define itself
If a node requires two roles, it has by
definition become a new role

Thursday, 7 February 13
A node can only have
one role
A role can include as many profiles as
required to define itself
If a node requires two roles, it has by
definition become a new role
Something couldnt be a lion and a
kangaroo at the same time!

Thursday, 7 February 13
It would be a Lingaroo

Thursday, 7 February 13
Roles

One-to-one to nodes
One-to-many to profiles
Only implement profiles

Thursday, 7 February 13
Example role
class role::uat_server {
include profiles::base
include profiles::customapp
include profiles::test_tools
}

Thursday, 7 February 13
Classification

Node classification simply assigns roles to


nodes
Roles expose profiles

Thursday, 7 February 13
Classification

node craig.puppetlabs.vm {
include roles::uat_server
}

Thursday, 7 February 13
Classification

Thursday, 7 February 13
The Stack

Resources
Thursday, 7 February 13
The Stack

Components: Resource modelling

Resources
Thursday, 7 February 13
The Stack

Profiles : Implementation

Components: Resource modelling

Resources
Thursday, 7 February 13
The Stack
Roles : Business Logic

Profiles : Implementation

Components: Resource modelling

Resources
Thursday, 7 February 13
Terminology

Profiles and Roles are Puppet modules


Components are Puppet modules
responsible for modelling resources
Everything is a module

Thursday, 7 February 13
Naming conventions
Components should be named after what
they manage (apache, ssh, mysql)
Profiles should be named after the logical
stack they implement (database, bastion,
email)
Roles should be named in business logic
convention (uat_server, web_cluster,
application, archive)

Thursday, 7 February 13
Hiera Overview
Lets talk about data!

Thursday, 7 February 13
Managing infrastructure

Dev

Thursday, 7 February 13
Managing infrastructure

Dev

QA

Thursday, 7 February 13
Managing infrastructure

Dev

QA

Production

Thursday, 7 February 13
Managing infrastructure

Dev

QA DC1

Production

Thursday, 7 February 13
Managing infrastructure

Dev

QA DC1 DC2 DC3

Production

Thursday, 7 February 13
Managing data in
Puppet is hard.

Thursday, 7 February 13
Without Hiera?
if ( $::environment == dev ) {
$ntpserver = 192.168.2.1
} else {
if ( $::fqdn == host4.mycorp.com) {
$ntpserver = 127.0.0.1
} else {
$ntpserver = 213.21.6.4
}
}

Thursday, 7 February 13
With Hiera?
$ntpserver = hiera(ntpserver)

Thursday, 7 February 13
Hierarchical lookups

Hiera uses facter facts to determine a


hierarchy
Top down hierarchy for overriding
configuration values based on roles,
environments, locations.... or anything else
And do this without any coding!

Thursday, 7 February 13
Separation of data from code

Puppet modules without hard-coded data


are easily shared and more re-usable
Infrastructure configuration can be
managed without needing to edit Puppet
code

Thursday, 7 February 13
Pluggable Backends
Source data from multiple locations
Data source is abstracted from code

Thursday, 7 February 13
Pluggable Backends
Source data from multiple locations
Data source is abstracted from code
hiera-gpg hiera-redis
hiera-http hiera-json
hiera-mysql hiera-zookeeper

Thursday, 7 February 13
Data Separation

Use Hiera to abstract your data from your


code
Components and profiles can source data
from Hiera

Thursday, 7 February 13
Profiles and Hiera

Use Hiera to model your data


Use profiles to model your implementation

Thursday, 7 February 13
The Stack
Roles : Business Logic

Profiles : Implementation

Components: Resource modelling

Resources

Thursday, 7 February 13
The Stack
Roles : Business Logic

Profiles : Implementation
Hiera:
Data
Components: Resource modelling

Resources

Thursday, 7 February 13
Classification

Assigning classes to a node


You can classify within Puppet code
(site.pp)
You can use an External Node Classifier
(ENC)

Thursday, 7 February 13
Leveraging an ENC
You can classify your nodes however you
want
Puppet Dashboard
Enterprise Console
Foreman
Site.pp
Custom script
Thursday, 7 February 13
Leveraging an ENC

An ENC should classify a node to its role


Nothing else

Thursday, 7 February 13
The Stack
Roles : Business Logic

Profiles : Implementation
Hiera:
Data
Components: Resource modelling

Resources

Thursday, 7 February 13
The Stack
Roles : Business Logic Classifier

Profiles : Implementation
Hiera:
Data
Components: Resource modelling

Resources

Thursday, 7 February 13
Key benefits
Reduced node-level logic to a role.
Gain the ability to be flexible with
implementation
Business logic improves managability by
non-Puppet users
Edge cases are now easy to solve

Thursday, 7 February 13
Enough Preaching!

Thursday, 7 February 13
This is not the way to
design Puppet... Its a
way.

Thursday, 7 February 13
Can I implement this
design without roles?

Thursday, 7 February 13
Can I implement this
design without roles?

Yes.
You lose the layer of abstraction that
exposes business logic

Thursday, 7 February 13
Can my roles be
defined in my ENC?

Thursday, 7 February 13
Can my roles be
defined in my ENC?

Yes.
Keeping it in code makes it versionable

Thursday, 7 February 13
Cant I just use Hiera
to define profiles?

Thursday, 7 February 13
Cant I just use Hiera
to define profiles?
Technically yes.
You lose the flexibility to implement code
logic in profiles and it may become
restrictive
You could possibly use: https://github.com/
ripienaar/hiera-puppet-nodes

Thursday, 7 February 13
The fundamental
concepts....

Thursday, 7 February 13
The fundamental
concepts....

Abstraction, abstraction, abstraction

Thursday, 7 February 13
The fundamental
concepts....

Abstraction, abstraction, abstraction


Decoupling business logic, implementation
and resource modelling.

Thursday, 7 February 13
The fundamental
concepts....

Abstraction, abstraction, abstraction


Decoupling business logic, implementation
and resource modelling.
Separating data and code

Thursday, 7 February 13
The fundamental
concepts....

Abstraction, abstraction, abstraction


Decoupling business logic, implementation
and resource modelling.
Separating data and code
Reducing node-level complexity
Thursday, 7 February 13
Other Resources

Adrien Thebos excellent blog post http://


sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as-
legos.html

My original blog post


2012/05/239/
http://www.craigdunn.org/

Module Structure Redux by R.I.Pienaar http://


www.devco.net/archives/2012/12/13/simple-puppet-module-structure-
redux.php

Thursday, 7 February 13
Thank you. Questions?

Follow me at @crayfishX
Bug me on Freenode: crayfishx
Enjoy the rest of Puppet Camp!
In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer
and helped me on my journey of discovery learning how to implement Puppet properly. R.I.P

Thursday, 7 February 13

You might also like