Jenni Griesmann, Senior Engineer
What are the most common tools that developers use at DigitalOcean?
While developers will inevitably use things like Git, Slack, Chef, or Jira, everyone uses whatever development environment they prefer. Others are always happy to suggest their favorite tools (sometimes biased toward things they may have created themselves).
Which languages do developers code in?
While there is a significant amount of code in the Go programming language, you’ll find almost anything if it will get the job done (Python, Ruby, Perl, JavaScript). There is always interest in finding the newest and best ways to solve problems, which means we often adopt newer versions and features as soon as they’re available and stable. Developers can also be found contributing toward open-source projects used by our services, so that means any language.
What is the development process like? (the lifecycle of a piece of committed code)
Generally, code changes are reasonably sized and scoped to make it as quick and painless as possible to go from an idea (which can come from anywhere) to a piece of code you can deploy out to the system. While there is a process for submitting a pull request, adding unit tests, passing integration builds/tests, and testing in a staging environment, there is a lot of trust given to developers to take responsibility for their own work and do what makes sense. If there is something slowing down the process or making things difficult, our developers have the power to change that.
What is code review like?
Code reviews (via GitHub pull requests) tend to receive prompt and useful feedback. My experience has been that people use it as a chance to point you toward things you might not be aware of, give you a quick proofread for typos or missed scenarios, and offer style suggestions without forcing them on you.
How is testing done, and what kind of tests are run?
DigitalOcean’s platform was designed to give developers a simple and elegant user experience with transparent and affordable pricing so they can spend time doing what they like to do—building great code. Because our platform is the very infrastructure for our community’s companies and projects, having solid, stable, and bug-free products is critical. Golang is very much designed around test-driven development and includes a very easy-to-use unit test infrastructure (automatically run on any push to the code repository). The goal is to allow anyone and everyone to run integration tests easily, so there are also a decent amount of preconfigured Docker images. As a final verification, DigitalOcean’s external API makes it easy to automatically test in either staging or a production cloud.
How is code deployed?
DigitalOcean embraces a continuous deployment methodology, which means changes are scaled smaller and deployed often. Depending upon scope and type of change, you might also see the use of feature flippers to gradually roll out changes if the extra effort to do so makes sense.
What is an average day-in-the-life of someone on one of the development teams?
Morning greetings in Slack
Quick check of messages
Digging into whatever development or testing you’re working on
Popping into a Google Hangout with a coworker to discuss code design or request background information about something you’re working on
Submitting a small pull request, merging, and deploying it
Joining a “munch and learn” (in person or over video conference) to hear about something a coworker may have been working on or experimenting with
Merging and deploying an update to the cloud—just not at 5 p.m. on a Friday unless absolutely necessary :)
What makes DigitalOcean a special place to be a developer?
Literally everyone has a voice. I have never heard or seen anyone’s opinions or suggestions dismissed or ignored outright. Whether the employee has worked here two minutes or two years, their perspective is valued because diversity in thought is exactly what makes a product better. This is particularly impressive considering the disparate locations many employees are working from (including myself—I’m based outside Chicago). No matter where you work, there are chat channels, video calls, and surveys you are never left out of.
Can you tell us a bit about your team and what you are working on?
I work on the hypervisor team where our Droplet offerings (cloud servers) are hosted, and it has given me a chance to dig more into open-source virtualization tools like QEMU and libvirt. I like knowing we can actively contribute to the platform our Droplets run on, as well as share some of our own projects (for example go-qemu and go-libvirt) with the rest of the open-source community.
Ines Sombra, Director of Engineering
What are the most common tools that developers use at Fastly?
We use GitHub, Slack, PagerDuty, Datadog, and more. If you’re a manager, you’re likely to be buried deep in the land of Jira.
Which languages do developers code in?
We are a polyglot company: We use Go, C, Ruby, Perl, Python, Ember, JavaScript… I believe our CTO is even writing a project in Rust.
What is the development process like? (the lifecycle of a piece of committed code)
When a pull request is issued against a repository, unit and integration tests are run in TravisCI. The changeset is then reviewed by someone else on the team; once comments have been addressed and CI passes, the pull request is approved and merged into master.
We then turn the changes into an artifact that can be deployed. Every team has the flexibility to tailor their deployments to their needs. For example, our UI uses a continuous deployment system. For the teams I oversee, we normally generate a package that gets installed with Chef.
What is code review like?
During code review, we look for test coverage, robustness of functionality, impact to the API, adherence to the patterns we follow at Fastly (we follow the Go code review comments), etc. We run go fmt and go vet as part of CI so all code that makes it to the master branch is known to be go fmt and go vet clean.
How is testing done, and what kind of tests are run?
We do unit testing, regression testing, and integration testing, all of which must run successfully before code can be merged.
For our Image Optimization team, Alice Nodelman (our wonderful QA manager) has constructed a custom testing harness that compares the similarity of images that our service generates with the ones generated by ImageMagick. This way we get confidence that features or changes we introduce do the right thing. We also run Alice’s test harness on every pull request alongside our unit tests. (Alice also did a talk with more about her testing methodology and how she approaches testing images without eyes.)
How is code deployed?
We deploy code using a few internal tools to coordinate and announce deployments, their risk, and schedules.
What is an average day-in-the-life of someone on one of the development teams?
I guess it depends on your team and your location. In the Edge Applications organization we have teammates in Portland, OR; Leeds, UK; and San Francisco, CA. I’d love to think remotes spend their days wearing pajamas. If you are in San Francisco you spend your days with me in the office surrounded by our cool office dogs.
Regardless of location you may work on new features, bug fixes, and/or developing new products. How your day looks depends on our team’s commitments for the week.
What makes Fastly a special place to be a developer?
At Fastly you get to be a part of the scaffolding that makes the web run. Fastly has a broad stack and very interesting problems. Our coworkers are experts on their fields (and nice people!), and we get to be curious and learn a lot.
Robert Speicher, Senior Developer
What are the most common tools that developers at GitLab use?
Git, GitLab, GitLab CI, Sentry, and Slack.
Which languages do developers code in?
Ruby for the monolithic Rails backend, JavaScript ES6 with Vue.js for the frontend, and Go for microservices.
What is the development process like? (the lifecycle of a piece of committed code)
The need for a code change or new feature is identified and discussed in the GitLab issue tracker for the relevant project. Before development begins, the issue goes through iteration by the product and UX teams to define expectations and requirements. From the issue, a developer will make the necessary changes in a Git feature branch, opening a merge request. The merge request will be reviewed by at least one other person before being merged. Depending on the priority of the merged change, it will either be released as part of a patch release or as part of the monthly release on the 22nd of each month.
What is code review like?
Every change goes through review by at least one “maintainer.” We have maintainers for frontend and backend, and if a merge request touches multiple areas it has to be approved by both a frontend and backend maintainer. In order to not create a bottleneck on maintainers, larger changes will often first go through a “reviewer” before going to a maintainer for final approval.
How is testing done, and what kind of tests are run?
GitLab has a large automated test suite that runs for every commit, including branches and forks. We test at the unit level, as well as the integration level via headless browser, and have a fully automated end-to-end QA test suite.
How is code deployed?
We deploy GitLab.com from the same packages that are available to users and customers, which means we don’t currently deploy multiple times per day, but rather as new versions are being prepared for release. We deploy the built packages across our staging and production fleets using internal tooling and Chef.
What is an average day-in-the-life of someone on one of the development teams?
We release a new version of GitLab every month, and each monthly milestone has several issues that we consider “deliverable” for that release, which can be new features, refinements of existing features, or foundational changes. Developers are assigned to those issues by the respective team leads. If a developer isn’t assigned to anything, they can pick any issue to work on, keeping in mind the issue priority, which is determined by the labels attached to the issue. For example, issues affecting customers have higher priority than others.
What makes GitLab a special place to be a developer?
GitLab is 100% remote so developers work when and where they want to. We’re also one of the biggest users of GitLab itself, so a large part of our developers’ jobs is actually improving the tools they use every day.
Yann Ramin, Software Engineer and Engineering Manager, and Jill Wetzler, Director of Engineering
What are the most common tools that developers use at Lyft?
Developers at Lyft are most at home with their favorite editor or IDE; we have a diverse crowd depending on what part of Lyft is being worked on. Backend developers will frequently use Emacs or Vim, but PyCharm and Gogland are popular for those who like a more complete development experience.
Lyft has a set of internal products named DevBox and OneBox, which let you run Lyft services in an environment that closely mimics a simple production deployment. DevBox runs on a laptop and allows for quick iteration, while OneBox is a persistent remote instance within Amazon Web Services capable of running the most important Lyft services all at once. These tools allow for complete end-to-end testing of backend code in conjunction with Lyft’s mobile applications.
We actively use Git and GitHub for code and code reviews; we manage hundreds of repositories for libraries and individual services. Jira is used for issue tracking as well as feature delivery.
Which languages do developers code in?
Most engineers are using Python and Go on a daily basis. Over the past few quarters we’ve been investing heavily in Go and Go tooling, and we’re seeing increased adoption as teams find out how easy it is to spin up a Go service. We give people a lot of boilerplate code for free—a basic service can be spun up and deployed in less than 24 hours, without much copy and paste. Data scientists often develop code in Python, R, Scala, and Java as part of their work, while several core infrastructure software components are written in C and C++.
What is the development process like? (the lifecycle of a piece of committed code)
It varies per team, but for the most part, engineers are either developing on DevBox or OneBox. Once they are happy with their change, they submit a pull request for code review to GitHub. After a code review :+1:, they can ship their code to master.
We built a tool called submitqueue, which manages the submission, revert, and testing of pull requests via a number of different emoji commands in the pull request comment section. For example, :rocket: to ship, :rocket: plus :praying hands: to ship without running tests, :loudly crying face: to revert, and :hammer: to rerun tests.
Once tests pass, changes are merged into master and are then ready to deploy. Teams generally deploy first to staging, then to canary, and finally to production. Teams are also encouraged to use feature flags in order to slowly ramp up their features, and turn them off in the event that their code introduces a bug.
What is code review like?
Code review is handled by each team on services they own using the GitHub pull request model. Developers push code to a branch in GitHub, and open a pull request. We have internal tools to mention specific developers using our internal “Please take a look” Slack bot. By default, anyone at Lyft can approve any change, and only one positive review is required to merge code into master for the next deployment. In certain cases, when a piece of code is sensitive, an “owner’s check” may be enforced, requiring review by a specific team or set of people (very little code at Lyft is behind an owner’s check). We also have automated bots dealing with simple syntax and styling issues, enabling developers to focus on higher-level discussions.
How is testing done, and what kind of tests are run?
We have a fair amount of manual testing that’s done by both in-house QA and outsourced QA via Testlio, but we believe strongly in automated testing. We have the following types of tests:
Unit tests, written by engineers and run on every change.
Security and linting checks, written by security and infrastructure engineers; these tests look for known security defects, hardcoded secrets, and other similar issues.
Integration tests, written by engineers and QA, run on every change. These tests check the interaction between services.
System tests, currently written by infrastructure engineers targeting certain core services.
Testing doesn’t stop in the test environment: A large amount of monitoring is built in to every service as part of our observability system. The system automates generating dashboards and alarms for staging and production environments while looking for common patterns and failures, freeing developers from having to reimplement the core monitoring of a new or existing service. The Observability and Infrastructure teams routinely push changes to the monitoring stack of every service to catch common errors and provide improved insight into a service’s health once it has left the dedicated test environment.
We routinely run other “live” tests, including tests of our system scalability, and fault injection to determine if we correctly route around isolated system problems in our microservices.
How is code deployed?
For most services, a deploy is started using Slack via our “deploy_bot”. Deploy_bot first lists any pending commits and offers quick links to see any deploys in action, and to kick off a new deploy. Each service has an independent deployment flow and can be updated independently of any other service. Deploy_bot acts as a gateway into a customized UI that steps the deploy through a number of targets, including updating their current development build (used by DevBox and OneBox), a staging environment, canary, and staged production deployments. Each step is accompanied by checks for the service successfully updating on each node in production. Teams running deployments use Grafana and Kibana to gauge the health of their code as part of the rollout. Deployment mistakes happen: Rolling back a set of changes is a single click, and often completes within one to two minutes.
A few services, primarily the old monolith, are deployed in a “train” model, where a certified conductor shepherds out a larger number of changes on behalf of others. This model is designed to reduce contention for deploys and to provide consistent monitoring by an experienced operator.
What is an average day-in-the-life of someone on one of the development teams?
Development teams can set their own working styles, but they’re usually structured as a mix of development time, a daily stand-up or SlackUp, and a weekly planning and triage meeting. Teams generally interact in person or over Slack to ask for code reviews, triage issues, coordinate development, or ask questions about development and deployment tools. Every team has a dedicated Slack channel—or a shared public help channel—which is used by engineers to get help on unfamiliar code or services. An engineer is usually active in multiple repositories, [and will often submit] pull requests [to ensure that changes across services are implemented seamlessly]. Other teams are very accepting of fixes and improvements.
What makes Lyft a special place to be a developer?
One of the core values we really live by is “Make It Happen.” There’s so much work to do! Engineers are encouraged to take on big projects, propose new features, fix persistent bugs, and take risks. Of all the companies I’ve worked for, Lyft probably has the highest rate of collaboration among engineers. People are helpful and willing to work in multiple repos.
Jonathan Barber, Engineering Manager
What are the most common tools that developers use at Rainforest?
Our development team is distributed around the globe, so our common tools are communication tools. We use Git (and GitHub) for source code management and reviews, CircleCI for CI/CD, email (Gmail), instant messaging (Slack), video conferencing (Zoom), Dropbox Paper and Google Docs for collaboration, and Google Calendar for scheduling when to talk (which is vital when your colleagues could be 12 hours [ahead of] you). Aside from that, our developers use whatever tools they are most comfortable with.
Which languages do developers code in?
Our backend stack is mainly written in Ruby (on Rails), although we have some services that deal with websockets written in Elixir, and some command line tools and simple HTTP-based services that are written in Golang. Our user-facing UI is written in JavaScript and uses React.
What is code review like?
Every merge to develop requires a review and approval from at least another member of the team for more complex changes individuals can ask for a second review. All questions that are raised by the reviewers must be answered. We have a lightweight technical review process that we do before we start coding to try to reduce the unexpected, or at least identify what we don’t know, and to avoid reinventing the wheel.
How is testing done, and what kind of tests are run?
Rainforest is a software-testing platform, so we work hard to have a fast, efficient testing process in place to make it easier for our team to succeed at shipping high-quality software. We have unit tests, integration tests, and QA tests (which are run using our own platform). All of the tests are run automatically as part of our deploy process.
How is code deployed?
We use GitFlow and CI/CD with CircleCI deploying to Heroku. Pull requests are developed on feature branches; after they are approved in review they are merged to develop. This kicks off a run of our test suites with our QA tests being run on a freshly initialized environment each time. Assuming everything passes, the develop branch is then merged to master and this deploys it to production.
What is an average day-in-the-life of someone on one of the development teams?
While our headquarters are in San Francisco, a large number of our employees (including me—I’m in Portugal) work remotely from around the world. Working remotely requires a certain amount of discipline; because the people you need might not be awake for another 12 hours, you have to plan ahead to make sure you don’t get stuck waiting for them, and [to ensure] that you don’t hold anyone else back. The flip side of this is that we’re really relaxed about what we consider a work “day” to be—as long as you meet the expectations of the team you can work however and whenever you want. Our developers frequently travel and take advantage of this flexibility, or take time to be with their family during the day.
What makes Rainforest a special place to be a developer?
I think we empower our engineers to make their own decisions and to question the tasks they work on, while giving them the support and time to allow them to succeed. Everyone in the company (not just the engineering team) is willing to help and to offer advice, which means that it really feels like a team.
Moiz Virani, Software Engineer
What are the most common tools that developers at Sauce Labs use?
In terms of tools, we use GitHub, Minikube, and code editors. I personally use Vim/tmux, Jenkins, Quay, etc.
Which languages do developers code in?
Python, JavaScript, and some Golang.
What is the development process like? (the lifecycle of a piece of committed code)
Our development process begins with a pull request in GitHub, then we run a variety of automated tests for feedback. Once the code has passed tests, each team plans and executes a deploy with their code.
What is code review like?
Our code review is in two parts: the peer review, which all stakeholders for the code participate in, and feedback from automated tests, which we run on each pull request. We use GitHub’s pull request mechanism for code review.
How is testing done, and what kind of tests are run?
We have a huge suite of automated tests that are run, including unit tests, lint checkers, integration tests, and end-to-end tests.
How is code deployed?
Different teams have different strategies for deploying code, but most get automated and deployed often. There are metrics for each team that indicate how often code is deployed.
What is an average day-in-the-life of someone on one of the development teams?
An average day might involve technical design meetings, scrum ceremonies, code reviews, and, of course, writing a lot of code.
What makes Sauce Labs a special place to be a developer?
Our unique technical challenges. Since we run and operate our own cloud, and we’re a PaaS, there’s a great deal of scaling and reliability challenges. Also, developers have the power when it comes to making important choices, so decisions are made after careful review of developer opinions and research.
Duretti Hirpa, Senior Engineer
What are the most common tools that developers use?
Far and away, the largest contingent of Slack developers work on our web application (Slack is incredibly committed to web technologies). As such, the most common tool used by developers at Slack (other than Slack itself) is Sublime Text—though there is a hearty contingent of people continuing the Emacs versus Vim debate. More developers are using VS Code as we transition from PHP to Hack. For our client engineers, Xcode and Android Studio are used. IntelliJ is popular as well.
As a group, we maintain a slack CLI that does common operations (ssh’ing into boxes, connecting to database hosts, installing dependencies, creating pull requests from the command line, etc.). My favorite subcommand (and one I authored) is slack beyonce—it opens a Beyoncé GIF in a browser window and reminds you that you, too, have 24 hours in your day.
Which languages do developers code in?
Engineers that work on the web application write PHP/Hack, HTML, Less, and JavaScript. Our backend services team writes Java, as does our Android team, but they’re starting to write some Kotlin now, too. Our iOS apps are written in Obj-C and Swift. Our Data and Search/Machine Learning teams write Scala. Our Automated QA team specializes in Ruby, and we also use it in the slack CLI mentioned earlier. The team that works on libslack (a business logic layer for our mobile clients) write C++. Calls (the feature that lets you screen share and make video calls in Slack) is written in Elixir. We even have Go in our stack for the distributed user model project (Flannel), as well as our service that issues universally unique IDs. We’ve got Python and Shell kicking around, too. We try to choose the right language, in context, for the job at hand.
What is the development process like? (the lifecycle of a piece of committed code)
The following is highly specific to developers that work in and around the web application (which I do). Our Data, Infrastructure, Native Clients, and Internal Tools teams have a similar but different development lifecycle.
Cut a branch from the main
webapprepository.Complete a discrete unit of work (fixing a bug, getting started on a feature). Decide if it’s going to be live or not—we have a set of feature flags that show and hide code in production. Write unit tests, or extend existing ones.
Open a pull request, and request a reviewer. Often, there is someone with context on the part of the codebase you maintain, but you can also select a random reviewer in Checkpoint. Checkpoint is Slack’s homegrown gating mechanism between our GitHub Enterprise instance and production. It also allows you to attach your local branch to a sandbox and deploy code. It was originally authored by our CTO, Cal Henderson, and is now maintained and extended by our Internal Tools team.
Once your code is approved—we require at least one approval before deploying—use Checkpoint to release your code. Under the hood, our web servers are on AWS and configured via Chef. We’ve got it such that AWS watches a key in Consul and downloads the correct version of the code from S3 when Checkpoint makes a modification (which means all our hosts get the latest code in just a few seconds). In practice, developers push a couple buttons and go about their day. We also have a Checkpoint bot that lets you know if a colleague deployed or staged your code (we ship a lot—approximately 100 times a day—so changes often go out in batches).
I often check our
#alerts-phpchannel after a deploy. We use Logstash to ingest our logs, and spikes in errors are cross-posted there. We also use Graphite metrics and create dashboards using Grafana; these help us determine rollout and impact. If the change goes awry, I figure out how to fix forward or revert my change.Optional: Socialize the change. This might mean telling your team or customer support agents, or telling a colleague in your triage channel, depending on the size of the change. If it’s a particularly impactful change, we post to
#ops, which is where we coordinate incident response and production changes. If it’s the end of a feature, you post in#released(our internal changelog channel)—it’s a great way to mark the end of a feature, tell the company, and thank everyone on a project.
What is code review like?
Code review is required at Slack, even in emergency situations. We take code review seriously, while endeavoring to have a humane code review process. (In fact, some of the earliest posts to our Engineering Blog are about good code review!) Code review helps you identify your blinds pots, prevent bugs, and learn from your colleagues. So much of development is couched in metaphor; code review is a perfect time to talk concretely about code in context. We also have dedicated channels where engineers can post their pull requests for review.
How is testing done, and what kind of tests are run?
Developers write unit tests for their code, and they manually test their changes. Every quarter we make a concerted effort to unit test more and more of our codebase, and to figure out how to test previously untested/untestable code. We are also heavily indebted to embedded automated QA engineers, who write incredibly valuable integration tests. We have two flavors of integration tests: smoke tests that run on every pull request, and a full battery of integration tests that are run several times per hour. There are also visual diff tests that help us detect unexpected differences or shifts in the UI. And we have a set of performance tests that run in order to make sure we aren’t adding bloat when we introduce new code.
How is code deployed?
Our homegrown system, Checkpoint, allows you to conduct code reviews, attach your local branch to a remote development instance, and stage and deploy your code. There are other Easter eggs as well, like mean time to deploy stats. Functionally, once your code is merged into master, you’re expected to confirm it works on staging. Once you’re satisfied, you hit a button and your code deploys to production. Our web servers are running the consul agent, watching and waiting for changes on a pre-defined key. When we click “Deploy,” Checkpoint updates that key in consul with the new path to a tarred up binary containing the current version of the webapp. This update triggers the watch on all the hosts, which then pull down that binary, validate its md5sum, and extract it into place. All of that happens on each host in just a few seconds, and there are significant checks and smart retries in place since it’s so critical to our workflow.
We’re a continuous deployment shop, and we send out hundreds of changesets a day. This means that it’s pretty rare that only one changeset goes out at a time. Checkpoint also allows us to see who’s authored what code. We have some unofficial rules around deploy: If you’re not “holding” a change (blocking a deploy while you check your work on staging), then anyone should and is encouraged to hit the deploy button. If that happens, you get a note from a corresponding Checkpoint bot (in Slack, naturally) letting you know which of your workmates is staging or deploying your changes.
What is an average day-in-the-life of someone on one of the development teams?
At Slack we write software in product teams. Product teams are cross-functional groups comprised of at least one of the following: a frontend engineer, a backend engineer, mobile engineers (iOS and Android), QA engineers (manual and automated), an engineering manager, a designer, and a product manager. Most teams also have a customer experience agent assigned. This CE agent acts as a customer advocate, and is very, very familiar with their pain points. These teams come together to work on projects vetted by Product. On any given day, you might be in the middle of a feature, bug fixing, on triage (when an engineer answers customer tickets as surfaced by our customer experience agents), or working on quarterly goals.
What makes Slack a special place to be a developer?
Slack is an incredibly supportive and collaborative place to write software. There’s a real feeling that while writing code can be a solitary act, making software isn’t. People are really kind and generous with their knowledge, and overwhelmingly willing to help.