Bitbucket - Lessons learned

@jespern

Tuesday, June 7, 2011 1

 Agenda

• Who am I?
• Bitbucket in summary
• Behind the scenes
• Lessons learned

Tuesday, June 7, 2011 2

History, acquisition, grown the team in SFO/SYD

How it’s built, technologies used, the evolution of tech
Lessons we’ve learned over the years, things that has helped us, and still help us day to day
Lessons learned will be the majority of my talk, will race through the rest
 Who am I?

• Founder of Bitbucket, now the “CTO”

• Started programming back in 1998
• Been using Django for 4 years (magic
removal branch, anyone?)
• @jespern on Twitter

Tuesday, June 7, 2011 3

Opening .exe files -> Basic -> PHP (.php3) -> C -> Perl -> Python
 Bitbucket in summary

• First commit was May 2008, launch ~1

month after that.
• 2 people, part time, for 2 years.
• Ran entirely on AWS.
• Acquired in August 2010.

Tuesday, June 7, 2011 4

Eirik left after 2 years, had a baby, just me after that.

Started from my blog.

 Bitbucket in summary

• Now 9 people, 6 engineers, one designer.

• 2 in Sydney, 7 in San Francisco.
• Only 2 people knew Django, only 3 Python!
• 1 is a Mercurial core developer.
• “Incubated” at Atlassian.

Tuesday, June 7, 2011 5

Mehmet and I knew Django, Brodie knew Python.

 Behind the scenes

• It’s all Python.

• 73934 lines of code, 86% is Django.
• We use Mercurial (surprisingly.)
• PostgreSQL, virtualenv/pip, etc.

Tuesday, June 7, 2011 6

The entire website is 100% django, only parts that aren’t are background stuff like scripts,
various SSH stuff, hooks.

Deploys semi-contained in virtualenvs, individual Pythons, most system-wide packages

inside the env as well (statically linked.)
 Behind the scenes

• On production:
• HAProxy, nginx, gunicorn, puppet.
• Roll out once a week, hotfixes go out
instantly.

Tuesday, June 7, 2011 7

Didn’t use to be like this, but more details on that later.

 Behind the scenes

• Testing:
• Selenium for functional tests.
• Kong for headless tests (@ericholscher)
• Bamboo for CI (just getting there.)

Tuesday, June 7, 2011 8

 Lessons learned

Tuesday, June 7, 2011 9

Experiences over the years, things that helped us and things that didn’t.
 Stay idiomatic

Tuesday, June 7, 2011 10

Probably the most important point I’m gonna make.

What I mean is to follow Django idioms, models, forms, don’t step too much outside the
common way of doing things.

Doing this has helped us *so* much. It’s not always the most intuitive thing to do, but it
comes in very handy in so many situations.
 From our codebase (actual code):

Tuesday, June 7, 2011 11

Basically everything’s like this, from the core concept of Repositories to everything else we
do.
 Everything’s an app

• Repositories
• Issue tracker
• User accounts (SSH, etc.)
• Compare view
• ...

Tuesday, June 7, 2011 12

They all have models.py, forms.py, views.py, etc.

Can perhaps run standalone but wouldn’t make sense.
 Benefits & Disadvantages

Tuesday, June 7, 2011 13

 • Free upgrades, little backporting
• Getting/finding help is usually easy --
awesome community

• Easy for new people to pick up...

Tuesday, June 7, 2011 14

...lead into where you can point them for the documentation (next slide.)
 “Where’re the docs?”
https://docs.djangoproject.com/

Tuesday, June 7, 2011 15

Really nice to be able to just point people there. They can go there and learn all the Django
idioms.
 • Kinda feels like you’re shoe-horning
sometimes

• Unusual application requirements (we’re

quite transactional)

• ..

Tuesday, June 7, 2011 16

CNAME is one thing we had to really twist Django into doing for us.

Transactional = involving the disk (we obviously do this a lot), handling errors, rolling back,
etc. The Django ORM isn’t *that* great. Certainly suffices, but does dumb things like
redundant joins and the transactional stuff (we use Postgres -- mention isolation levels.)
 We stick to idioms
For better or worse.. Usually better.

Tuesday, June 7, 2011 17

Next up, things change.

 Things change

Tuesday, June 7, 2011 18

Don’t do vendor lock-in! We did (AWS - S3/cloudfront, but mostly EBS), and it hurt is quite a
bit down the line. The places in which we didn’t, came in *really* handy.

We took the pain of making sane technology choices and decisions early -- often a good
idea. AWS great for bootstrap, but we didn’t make use of anything we couldn’t replace later
(EBS looks like a disk).
 Embrace it!

Tuesday, June 7, 2011 19

Design your app in such a way that you’re not bound to any specific technology or vendor.

Changes *will* happen, whether it’s to replace a bad technology, or more often, to make
something scale better.
 Good choices we made

• Linux

• Django

• WSGI

• No need to pick a DB

• Everything else we left

open-ended

Tuesday, June 7, 2011 20

WSGI was awesome, as there’s a ton of WSGI hosts out there -- we’ve been through
mod_wsgi, uwsgi, and now gunicorn. Hell, we even ran FastCGI.
We could rip out postgres for mysql or whatever any day.
 Open source

Tuesday, June 7, 2011 21

Piggyback on existing efforts, stand on the shoulders of giants. Don’t fear NIH.
 We use stuff like...
• Celery, South, etc.

• -(social)registration,
-compressor, ...

• A lot of really well-done

stuff out there!

Tuesday, June 7, 2011 22

 And try to give back
• django-piston

• django-dogslow

• tipper

Tuesday, June 7, 2011 23

Explain what dogslow is, how it relates to tipper.

 Be transparent
& humble

Tuesday, June 7, 2011 24

Explain outages, don’t think you’re better than your users.

 Thanks!

Tuesday, June 7, 2011 25