Tuesday, October 08, 2019

CAYL - Code as you like day

Building an enterprise grade distributed service is like trying to fix and improve a car while driving it at high speed down the free-way. Engineering debt accumulates fast and engineers in the team yearn for the time to get to them. A common complaint is also that we need more time to tinker with cool features and tech to learn and experiment.

An approach many companies take is the big hackathon events. Even though they have their place, I think those are mostly for PR and getting eye candy. Which exec doesn’t want to show the world their company creates AI powered blockchain running on quantum computer in just a 3 day hackathon.

This is where CAYL comes in. CAYL or “Code As You Like” is named loosely on “go as you like” event I experienced as a student in India. In a lot of uniform based schools in Kolkata, it is common to have a go as you like day, where kids dress up however they want.

Even though we call it code as you like, it has evolved beyond coding. One of our extended Program Management team has also picked this up and call it the WAYL (Work as you like day). This is what we have set aside in our group calendar for this event.

“code as you like day” is a reserve date every month (first Monday of the month) where we get to code/document or learn something on our own.
There will be no scheduled work items and no SCRUM
We simply do stuff we want to do. Examples include but not limited to
  1. Solve a pet peeve (e.g. fix a bug that is not scheduled but you really want to get done)
  2. A cool feature
  3. Learn something related to the project that you always wanted to figure out (how do we use fluentd to process events, what is helm)
  4. Learn something technical (how does go channels work, go assembly code)
  5. Shadow someone from a sibling team and learn what they are working on
We can stay late and get things done (you totally do not have to do that) and there will be pizza or ice-cream.
One requirement is that you *have* to present the next day, whatever you did.  5 minutes each

I would say we have had great success with it. We have had CAYL projects all over the spectrum
  1. Speed up build system and just make building easier
  2. ML Vision device that can tell you which bin trash needs to go in (e.g. if it is compostable)
  3. Better BVT system and cross porting it to work on our Macs
  4. Pet peeves like make function naming more uniform, remove TODO from code, spelling/grammar  etc.
  5. Better logging and error handling
  6. Fix SQL resiliency issues
  7. Move some of our older custom management VMs move to AKS
  8. Bring in gomock, go vet, static checking
  9. 3D game where mommy penguin gets fish for her babies and learns to be optimal using machine learning
  10. Experiment with Prometheus
  11. A dev spent a day shadowing dev from another team to learn the cool tech they are using etc.
We just finished our CAYL yesterday and one of my CAYL items was to write a blog about it. So it’s fitting that I am hitting publish on this blog, as I sit in the CAYL presentation while eating Kale chips

Sunday, September 29, 2019

Azure Dedicated

I remember a discussion with a group of friends around 8 years back. Microsoft was in it’s early days of becoming a leader in the cloud. Those friends, all techies in the Seattle area had varying expectation on how it would work out. Many thought that a full blown move was few decades away because their experience indicated that all big companies ran on stack that was very old and simply couldn’t be moved to the cloud any time soon.

Waves of workloads have been since moving to the cloud. A new brew of startups were cloud native from the start and they were the first to use the power of cloud. Many large and small enterprises had already virtualized workloads and they moved as well. Some moved their new workloads (green-field), some even followed lift-n-shift with some modifications (brown-field) into the cloud.

However, a class of large enterprises were stuck in their data centers. They wanted to use the power of the cloud, they wanted to use IoT integration, Machine-learning and the capability of elastic growth of their applications, but the center of their systems were running on some stack that did not run in the standard virtualization offered in the cloud. These enterprises said that if they cannot move those workloads into the cloud, they would need to keep the lights on in their data-centers and moving some peripheral workloads simply did not make sense.

This is where Azure Dedicated and we come into the picture.

SAP HANA Large Instance

For some of these customers that #$%#@ is SAP HANA in-memory DB on a single machine with 768 vCPUs and 24 terabytes of ram (yup) and we have them covered. Some wanted to scale those out to 60 terabytes in memory, we have them covered too with our bare-metal machines running in Azure. See SAP HANA Large Instances on Azure.

We started our journey in this area with this workload. Now we have evolved into our own little organization in Azure called Azure Dedicated and also support the following workloads.

Azure VMware Solutions

Some customers wanted to run their VMware workloads and we have Azure VMware Solution for them.

Hardware Security Modules

In partnership with other teams in Azure we support HSM, which are standard cryptographic appliances powering say financial institutions.

Cray Supercomputer?

So you need to simulate something or do ML on tens of thousands of cores, we have Cray super computers running inside Azure for that!!

Azure NetApp Files

Working closely with the storage team we deliver demanding file based workloads running on Azure NetApp files


In partnership with SkyTap we provide IBM Power workloads on Azure to customers.

What next?

As an organization we are always on the lookout for more exciting opportunities to bring the power of cloud to our customers, wherever they are. So keep looking out for new announcements from our org.

How do we support these workloads?

Our team builds cutting edge, highly available, scalable, distributed systems to power these workloads in Azure. Our engineering stack uses linux based containers on Kubernetes, AKS, Helm, linkerd, fluentd, docker, go-lang, python, .netcore, C#, F# etc. Some samples of how we build and run our systems