Recently, both at Pixmat and at IFARHU we have had to plan the launch and execution of some software projects that we have been working on for some time.
In my concept, the deployment (deployment) does not begin at the moment in which we execute the code in the environment, but is likely to be one of the most advanced phases of this process, which should begin with the planning of the architecture of our application.
In this post I will try to break down some good practices that I have been able to compile from my own experience, which may be useful to many.
#.1. Always make sure to use a Version Control System
This point is quite obvious, but it is still important to remember. Use Git or any other version control system (CVS).
Git is the most used today and there are systems, based on Git, that make your life easier, such as:
#.2. Twelve-Factor App
As I said initially, deployment should be seen as a process, where the system architecture plays a role within it.
Twelve-Factor App is a “methodology” for manufacturing SaaS-type applications. I see it more as a series of recommendations, as indicated in the original text:
"Our motivation is to raise awareness of some systemic problems we've seen in modern application development, to provide a shared vocabulary for discussing those problems, and to offer a set of broad conceptual solutions to those problems with accompanying terminology."
It consists of twelve principles, which I state and give an example for each of them:1. Codebase: Maintain a branch (see point 1) for each environment. I would even recommend implementing some branching system like git-flow.
- Dependencies: Use Maven (Java), NPM / Yarn (JavaScript), Composer (PHP), Bundler (Ruby) or any other similar system, but separate the dependencies from your source code. Always.
- Config: The configuration should not be in the repository, ever. Do you have the database passwords in Git? You started badly. Use
dot env(available in several languages) or use environment variables. - Backing services: The added services (Database, a Message Queue, etc.) must be treated as independent services (but linked to the project, as if they were dependencies).
- Build, release, run: Maintain the well-structured deployment process, in addition to different environments (for different phases of the project).
- (Stateless) Processes: Applications should not have states or share them with other parts of the application (that is why Databases should exist, for example). The application should be viewed as a single process.
- Port binding: The application must be “self-contained”. That is, they must use a protocol (such as HTTP) to become public, by assigning a specific port.
- Concurrency: Always think about the scaling of your application (we will see some points about this later).
- Disposability: Make sure that if your services restart, something doesn’t explode. A very common mistake is to execute very heavy tasks in the foreground, through the HTTP protocol (when there are other more correct ways to do it).
- Dev/prod parity: Development, stage, production should be as similar as possible. This helps to detect possible failures or even to be able to replicate them. The most important thing is that you can test with a system that is as close to production as possible.
- Logs: Another very common mistake is not treating logs as streams of events. I will address this later.
- Admin processes: Read the Disposability point. This is one way to correct the given example. Processes must run outside/independently.
#.3. Make sure you have tests (and automate them)
Another fairly obvious element, but it is important to keep in mind, is that your system must have different layers of automated testing.
The scope of the tests is the subject of a totally different post, but I think it is also important to emphasize that:1. Have different environments, as indicated in the previous point, for different phases of the project.
- Have at least three environments: local (development), testing, production. You can have more, but at least make sure you always have these three.
- All environments should be as similar as possible. I would even say that they should be as similar to production as possible. Today with tools like Docker, Kubernetes or Vagrant it should be impossible to say “it works on my machine”.
- Never, never, but never. Never send anything to production that hasn’t been exhaustively tested in the other environments.
- Don’t rush to deliver to production.
#.4. CI/CD
In addition to the previous point, it is useful to have a Continuous Integration (CI) system and a Continuous Delivery/Deployment (CD) system that facilitates the process of performing automatic testing and automatic deployment to different environments.
There are different tools that can help you with this, I would recommend Jenkins, Bamboo and GitLab.
#.5. It can fail. Are you ready?
As the Aerospace Engineer, born in Panama (Zonian), Edward Murphy Jr. said in the laws that bear his last name: “If something can fail, it will fail”.
The problem for many is not only that it fails, the problem is that we must be prepared for it.
- Remember the logs, for everything (with emphasis on what can go wrong).
- If it fails, make sure you collect as much information as possible.
I recommend using tools like Splunk, New Relic, Sentry or BugSnag.
#.6. Would it scale horizontally? Centralize logs, cache and sessions, files
Always think that your system can or will grow and when this happens, you must ensure that the system allows growth.
Be sure to comply, more importantly, with some points covered in the Twelve-Factor App, when centralizing parts of the system.
For those unfamiliar with the term “scaling out,” it means that we need to make sure we can have different application servers that can distribute the load between them (usually with a load balancer, like HAProxy, in front).
###Logs
Don’t save logs to files when you can treat them as event sources.
In addition to the tools to detect failures in your system, described in the previous point, I can recommend another tool that we use a lot at Pixmat and it is GrayLog.
GrayLog allows you to centralize and manage all logs, from different platforms and systems. Through this platform we usually monitor the logs of:- Web Server (Nginx or Apache)
- Database Server
- Systems / platforms logs
#.Cache and Sessions
Do not save the session to files only within the server. The problem with this practice is, if you scale out, your sessions will only remain on one of the servers and could not be used by the rest of the servers. The same would happen for the cache.
The simplest solution for both cases is to use Redis. At both Pixmat and IFARHU we use Redis (both AWS and Azure, respectively, have Redis services).
We also use Redis as message broker, although in some cases we use specific services such as RabbitMQ.
#.Files
If your application uploads files (photographs, reports, or others) or has files that require use in all applications, consider centralizing a single source of files.
This point is important, since if we have different application servers, the file could only remain on one of them (where it was uploaded). Obviously, we can mitigate this if we save the full path of the file (and its reference to the server) and expose them through specific domains/subdomains, but it is not a clean or adequate solution.
The best solution is to use services like Amazon S3 or Azure Blob Storage which are inexpensive and maintenance-free.
Also consider using a CDN, such as CloudFront.
#.7. Remember to monitor
Remember that there are things that can go wrong and it is important to be prepared. To do this, you must monitor services and consumption (both resource and money, if you use the Cloud).
Here you have different alternatives. In the case of Pixmat, our stack is usually using Prometheus and Grafana, but you can use services like CloudWatch or Azure Monitor.
#.8. Has your app grown or do you have spikes in usage? Remember the Cloud
At IFARHU we use Azure and at Pixmat we use AWS more for certain applications. The great advantage of the cloud is the almost transparent scaling of resources according to need or availability.
Another great advantage is that you can have different (identical) environments immediately, which would facilitate the testing process or even allow you to deploy temporary environments just to test specific features or stories.