I was troubleshooting an interesting bug recently thanks to which I've learned a bit more about Hangfire.io and expressions in .NET.

The situation was that Hangfire dashboard looked correctly, we had all jobs registered as expected. But what was actually executed by the scheduler for each job was same logic, which was supposed to be executed only for the last job. Nasty bug. We were not yet on production with hangfire.io, but still it was quite an unexpected behavior to see.

Reason was that we were wrapping each job in a class called JobRunner. This class was adding some generic functionality to update UI progress bars when jobs are running. Our code looked like that:

JobRunner runner = new JobRunner(myJobClass);
RecurringJob.AddOrUpdate(myJobClass.JobId, () => runner.Execute(), myJobClass.CronExpression);

Crucial thing to understand about Hangfire is that the what we pass to AddOrUpdate method is not a function to execute but an Expression describing the function to be executed. See this thread for difference between Expression<> and Func<>.

runner instance is not kept in memory or serialized. When Hangfire executes the job, it needs to create the instance by calling the constructor of given type. Constructor arguments are resolved from IoC container. In our case constructor argument was of type IJob. This interface was providing properties like JobId or CronExpression. So what was happening when EVERY job was running, was firsts implementation of IJob found in the container injected into a JobRunner. For each job same implementation of IJob was injected. And here we are - all jobs magically are executing same logic...

Now it seems quite obvious but it was necessary to learn couple of rules along the way to understand that behavior. It seems to be a common misunderstanding as there is even a comment about people making that mistake in hangfire.io source code, see Job.cs .

I hope this case study will help someone to avoid similar traps.

Building software is expensive. Software developers and other necessary professionals have high salaries. Coding is time consuming. We all know that.

So how we can save money? By hiring cheaper developers? By hiring developers who code faster? By working longer hours? Well, not really, those solutions, if even rational, are only about moving costs from one place to another.

Biggest savings we can get by strictly controlling the scope of the project.

There is too much waste in the industry.  Many teams build features that nobody uses. Developers are solving abstract problems that do not really exist from business domain perspective. Business is asking for things that never convert to added value. Software should be an investment but often is only a cost.

How to prevent that? It's hard, but here is my advice:

• Think at least 3 times if the idea converts to value before starting building. Ask for feedback. Gather data. Starting a project based solely on intuition is a risky business. Without solid data, it's like gambling. Money can be lost in this game, so be aware of how much risk you can accept.
• Ok, so you have a proven arguments that the idea is wort building. Can it be achieved simpler? Maybe there is already a tool that you can use? Affordable SaaS platform? Open-source software? Think 3 times about how to make it efficiently.
• If the project is not a standard thing that you did already - start small. Do the research. Build proof of concept of the unknown parts. Define solid technical foundation for the project. It will be harder to change it later. Check what works and what doesn't in your case.
• Do not involve big development resources before you know what you expect from them. Prepare the project. Know your budget. Know your timeline. Check with technical people if the assumptions are realistic. Make at least a rough estimate. Think about the risks that may impact the estimate. Are the risks acceptable?
• Verify the results often and early. There should be something usable produced as soon as possible. Start from the core business. Do not build "nice to have" things when core functionality is not ready. Nice to have things may kill the project. Be strict about the scope and constantly challenge it. Is this story or sub-task worth doing? Asking that question is not a signal of laziness, it is a signal of caring about the budget, the timeline and the impact.

Based on my experience, especially from start-up and innovative projects, those are the rules that are often forgotten. We tend to rush, but as a result, instead of being quicker we may be slower because of loosing the correct way. We want to be agile but we forget about planning and create too much chaos instead of agility. We say that we care about quality of the product so we build the whole plastic toolbox instead of just one precise tool at time. We let engineers to own the estimates but we are not communicating the constraints and not controlling the scope.

Those are the major sins that I try to address by the above rules. I hope that you will find it useful also in your project.

It I was an interesting finding this week. I was not aware that it is possible to use multi-line C# code snippets inside Azure API Management policies.

For example if you'd like to calculate an expression (could be e.g. a backend service query parameter) based on a request header value, that you could use a snippet similar to this one:

@{ 
  /* here you can place multi-line code */ 
  new dict = new Dictionary<string, string>() {
    {"a", "1"}, 
    {"b", "2"}
  };
  var val = context.Request.Headers.GetValueOrDefault("test", "a");
  return dict.ContainsKey(val) ? dict[val] : "1";
}

Details about expressions syntax can be found here: here: https://docs.microsoft.com/en-us/azure/api-management/api-management-policy-expressions

One gotch'a was with moving that kind of policy definition to Terraform.  It is necessary to replace characters "<" and ">" with entities: < and > respectively. Otherwise Tarraform could not apply the changes, although it worked directly in Azure portal.

Worth to note that you could achieve the same by using control flow policies, but this example is only an illustration, you can have more complex snippets e.g. for request verification or composing/manipulating response body.

I've been doing some research recently about architectures for large scale data analysis systems. An idea that appears quite often when discussing this problem is lambda architecture.

Data aggregation​

The idea is quite simple. Intuitive approach to analytics is to gather data as it comes and then aggregate data for better performance of analytic queries. E.g. when users do reports by date range, pre-aggregate all sales/usage numbers per day and then produce the result for given date range by making the sum of aggregates for each data in the range. If you have let's say 10k transactions per day, that approach will create only 1 record per day. Of course in reality you'd probably need many aggregates for different dimensions, to enable data filtering, but still you will probably have much less dimensions than the number of aggregated rows.

Aggregation is not the only way to increase query performance. It could be any kind of pre-computing like batch processing, indexing, caching etc. This layer in lambda architecture is called "serving layer" as it's goal is to server for the analytical queries as a fast source of information.

Speed layer​

This approach has a significant downside - aggregated results are available after a delay. In the example above the aggregated data required to perform analytical queries, will be available on the next day. Lambda architecture mitigates that problem by introducing so called speed-layer. In our example that would be the layer keeping data for current day. The amount of data for a single day is relatively small and probably does not require aggregates to be queried or can fit into a relatively small number of fast and more expensive machines (e.g. using in-memory computing)

Queries​

Analytical queries combine results from 2 sources: aggregates and speed layer. The speed layer can be also used to crate aggregates for the next day. Once data is aggregated it can be removed from speed layer to free the resources.

Master data​

Let's do not forget that besides speed layer and aggregates, there is also a so called master data that contains all raw, not aggregated records. This dataset in lambda architecture should be append-only

Technology​

This architecture is technology-agnostic. For example you can build all the layers on top of SQL servers. But typically a distributed file system like  HDFS would be used as master data. MapReduce pattern would be used for batch processing the maser data. Technologies like Apache HBase or ElephantDb would be used to query the serving layer. And Apache Storm would be used for the speed layer. Those choices could be quite common in the industry but technology stack can vary a lot from project to project or company to company.

Sources​

http://lambda-architecture.net/ https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude https://databricks.com/glossary/lambda-architecture

In software we like things to be generic and reusable. When designing data models we try to predict the future and possible scenarios where system can evolve. We often forgot that to be reusable, first system must be usable.

I like generic mechanisms. Generic search engine, generic RPC library, generic audit, generic authorization framework, generic UI components. Reusability on technical level is great. You can quickly build new things from existing blocks.

But generic business model? It is much easier to work with business domain model when it is focused and very specific. There should be no properties, classes, database tables or relationships just in case for the future. Every domain class and field must have specific, well defined meaning and purpose. There should be a real-life story behind each domain element. There should be no place for misinterpretation. Source code is often the most reliable source of truth about how whole organization works. Source code and its data must not lie about what it can really do.

I like to design systems by starting with business use-cases, end-to-end user flows and data flows. Once you have those dynamic elements of the system defined, then data model to support the processes should be straightforward. It is the business process to define domain model, not the other way around.

Development teams tend to focus too much on generalization to support unknown processes instead of pushing stakeholders to define concrete processes. Let's do not pretend that we are prepared for all possible future scenarios. Both in business and in code. Let's focus on bundling something that works and is well defined. This is why we adopt agile methodologies - to be able to evolve our models in iterations. This is why we adopt microservices - to be able to retire outdated process easily and build new ideas independently.

To conclude. Avoiding generalization on business domain level is good. It is not an anti-pattern. Generalization is often a distraction from the real goals. Generalization makes the goals less achievable and vague. Let's be focused and specific.

I've share in one of my articles that is is estimated that only 10% of the learning happens a the formal training. The remaining 90% comes from everyday-tasks and learning from coworkers. Formal training includes things like self-studying from online resources which I would like to emphasis in this article. Since it's only 10%, can it be treated with low priority?

Self study - 10% of the time​

I do not know how those numbers were calculated. I've seen those numbers in on of the managers training. In my opinion those numbers can be quite accurate when we think about the time that we are able to spend on learning. Most of the time we spend at work and this is where we have the biggest opportunity to learn. Work builds real experience and practice. The knowledge becomes not just theoretical but also tested in real-life. We are able to come up with our own use-cases, examples and experience practical challenges.

Studying vs practicing vs teaching​

When we think about the levels of knowledge it may be illustrated as: student → practitioner → teacher. Self-study brings you only to a student level. Good courses include hands-on labs, so that you can experience also some practice. But training exercises are always simplified and cover only simple "happy paths". They do not include production-level challenges. It's like fight with a shadow vs fight with a real opponent in martial arts.

Self-study - the impact​

But does it mean that self-study can be ignored as it contributes only 10% to your learning? Absolutely not. It's 10% in terms of the time but can be much more in terms of the impact. We do not always have the comfort to learn new things at work. Especially when you are an architect or in general technical lead, your company expects that you are the one who teaches others, who knows the new trends and who is up to date with latest technologies. You have to do a lot of self-study so that the whole company does not settle down with old solutions.

Online resources for self-studying​

I was recently writing about DNA program which is a great example of self studying materials for software architects. Recently I've also singed-up for Google Cloud Platform Architecture training with Coursera. After doing the first module and getting this certificate I can definitely recommend Coursera GCP trainings. The best thing is that the training include a lot of hands-on labs with real GCP resources provisioned via Qwiklabs platform.

I have to admit that the online training possibilities that we have now are amazing. For a small price you can have access to resources that are often of much better quality than (unfortunately) some lectures at stationary universities.

Summary​

So, go and self-study! Then use it at work and build a better world :)

I was starting my career in software development in 2008. We are now in 2020. What seemed to be true in 2010 and seems funny in 2020?

• jQuery is the best thing that could happen for JavaScript and best library ever
• Internet is not usable on phone. You need a desktop browser
• You need to be a Linux or Windows guru to setup a production-ready server
• SQL database is the most important component of the system. You cannot just store data in files
• We do not have any unit tests and are not worried at all
• Agile methodologies is a new thing. We have read about it and maybe will try in a new project after the specs are ready
• Big data is a new thing and it's about creating OLTP cubes
• Machine learning is a niche academical subject, not usable in practice
• Functional programming is a niche, good only for mathematicians
• XML is nice, you can use cool XSLT transformations with it
• You cannot be hired as a programmer after 3-weeks course
• You can use cookies and users do not have to know

In 2010 you could say those things and be promoted to a team leader as a very wise person.  How does it sound in 2020? :)

To get even more sentimental you can also check this article as a nice summary of last decade in software development: https://blog.pragmaticengineer.com/the-decade-in-review-in-software-development/

The aim of this article is to build a high-level mind-map and understanding of concepts like Kubernetes, Helm and cloud-native applications. The assumption is that you have worked already with Docker.

Docker​

So you know Docker. Docker container is like a virtual machine but lighter. Why lighter? It does not have operating system inside. It relies on host operating system. Docker adds only applications layer on top. You can run many Docker containers in a single server / virtual machine.

Docker Compose​

You may have also heard about Docker Compose. For example when your application consists of .NET Core web server, MySQL database, and Redis cache - you can define 3 separate containers for it. To run all of them in a virtual network you define a docker-compose.yml file. Then 3 of them can be run with a single docker-compose up command.

Scaling the application for production​

Now let's imagine you want to scale your application. You introduce a load balancer and 2 additional web server containers. You are also adding a RabbitMq container and an instance of a background processing worker. There are also other requirements for production environment:

•  containers need to be distributed across many servers
• containers which do not need much resources can be run together in same server to use provisioned servers in a cost-effective way
• when a container is not responding it should be restarted
• when connectivity with container is lost it should be replaced with a new instance
• number of containers should autoscale
• number of servers should autoscale
• new containers added to this environment should be auto-discovered
• it should be possible to mount and share storage volumes in a flexible way

Things are getting complicated. To achieve all that requirements we must write a lot of code to monitor and manage infrastructure. This is called containers orchestration. Or... we can use Kubernetes (k8s) which has all that features and more built-in!

Kubernetes concepts​

Pod

Abstraction of a single app. It can have one ore more containers. If containers are tightly coupled they may be placed in same pod. All containers inside a pod share storage volumes. A pod is an unit of deployment and scalability. Each pod has IP address assigned, so there is no need to care about port conflicts.

Node

This is how k8s names physical servers or virtual machines hosting the containers.

Cluster

Set of nodes available for k8s. Example cluster would be 4 nodes and 20 pods running on them, managed dynamically by k8s.

Namespace

All objects within a cluster can have a namespace. It allows to create many virtual clusters inside single cluster. It is useful for example to model many independent environments for staging in a single k8s environment.

Service

Since each pod has it's own IP and pods can be started and shut down, it would be not easy for other pods to keep track of constantly changing IPs. That's why we have Services in k8s. Service groups a set of pods by given labels. Pods may come and go, but as long as labels criteria are matching, all matching pods are automatically tracked. Service has a logical name assigned, so that other pods can use this name to communicate with the pods behind the service. Service routes and load-balances the traffic dynamically to relevant pods.

Services can be also used to point traffic to an endpoint outside k8s cluster. In this case instead of defining a service by providing pod labels selector, it is necessary to define an IP of the service backend.

Ingress

Ingress is used to expose services to outside word via http(s). It can also terminate SSL.

Deployment

Deployment specifies the pod and the number of its replicas that should be run. Deployments controller is responsible for rolling out updated pods (e.g. with updated container image). It starts new pods, shuts down old pods and then keeps monitoring them to make sure that desired number of replicas is run.

StatefulSet

StatefulSets are used to manage containers which contain data. Containers that have data cannot be just removed and replaced as we cannot loose its data. Pods in StatefulSets have sticky identities and persistence storage assigned. Persistent storage is not deleted when pod is deleted.

Worth to mention that in many scenarios managing persistence would be simpler outside Kubernetes cluster. Many cloud providers have sql an noSql as-a-service offerings which usually takes care about things like backups, availability and replication.

Monitoring containers​

Each container has a liveness and readiness probe defined. Typically those are HTTP endpoints called by Kubernetes to check if container is healthy. K8s calls the endpoints periodically, e.g. every 10 seconds depending on configuration. When liveness probe fails container is restarted. When readiness probe fails traffic is not anymore routed to this instance. Health check endpoints must be implemented in every service. Simple liveness endpoint could be just returning status code 200. Readiness endpoint could additionally check things like database connection, cache readiness or amount of currently used resources to check if service is really ready to process new requests. When readiness endpoint detects a problem that could be solved by restarting the container, it could potentially switch a variable to force liveness endpoint to fail causing a restart.

Helm​

An application targeting Kubernetes is configured in a set of yaml files for deployments, services etc. Those sets of yaml files can grow pretty complex. We also need some versioning tool for them. A common approach is to package all k8s files into a Helm package called a chart. What is being deployed to k8s cluster is a chart.

Cloud native applications​

Kubernetes is an operating system for cloud native applications. Cloud native applications are usually designed in microservices architecture and aim to be cloud-agnostic - can run in many public clouds or in a private/hybrid cloud. There is already a lot of predefined Helm charts available in public repositories if you'd like to pull scalable k8s setup e.g. for Cassandra or Prometheus in your cloud-native setup.

Sources​

• https://kubernetes.io/docs/home/
• https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes
• https://medium.com/kubecost/understanding-kubernetes-cluster-autoscaling-675099a1db92
• https://helm.sh/docs/
• https://platform9.com/blog/kubernetes-helm-why-it-matters/
• https://thenewstack.io/10-key-attributes-of-cloud-native-applications/

When working with projects using microservices architecture I opt for monorepo pattern. A single repository containing source code of all the services. It facilitates knowledge sharing between teams and encourages more unified programming style. Microservices give a lot of technological freedom, but this freedom should be used wisely. Common standards across the organization are still important.

Working with a single repository forces all the programmers to at least have a glance at the list of commits from other teams when making git pull. This channel of knowledge sharing can be very beneficial.

Despite having a single repository, we still need separate pipelines and policies for separate directories. Azure DevOps facilitates it by allowing directory path filter in crucial places:

• path filter in build trigger settings
• branch policies
• required reviewers

It allows to setup a clear ownership of different parts of the repository and apply different pipelines to different parts of the repo while still having all the benefits of monorepo pattern.

What are the most important things when building software? Is it architecture? Frameworks? Process? No. Those are only tools.

What really matters is the fact of delivering useful solution. Usefulness defined in terms of number of users or revenue. Useful solutions are not delivered too late or too early - they are delivered on time. Useful solutions are not too slow or too buggy - they have acceptable performance and are reliable.

What also matters is the satisfaction of the team after building the solution. It could be that kind of satisfaction which painter has when working on a painting - satisfaction during the process. There can be also some pain in the process but that would be another type of satisfaction - the one that athlete has after finishing a marathon. It was painful but feels so good after the finish. It could be also like a satisfaction of a cook who has just discovered new spices and learned something useful for the future.

Studying architecture or frameworks is also important, because it makes us efficient and more likely to deliver useful solutions. But the goal is not to learn, the goal is to apply the knowledge and make things happen.