tag:blogger.com,1999:blog-64346752024-03-15T18:10:05.550-07:00Bong Geek - Abhinaba BasuBong is an affectionate slang for people from Bengal (where I am from)Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.comBlogger223125tag:blogger.com,1999:blog-6434675.post-4494773465352313262022-11-28T07:42:00.001-08:002022-11-28T07:42:00.211-08:00Wordament Solver<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi3JcuuAO0hezR91BjbWiv4SJ6mtCuVYQtIHXFS9b2smnEtMoVPYbGL4W5wIpq8geopeib6cAQNT2_tkykuovlEsc_FLQzyf_hWZ3GqHLasC50pMkQB-fDuPpELGEZPQtzkVqDRUPWtW8qCzH6Yw4PYCk7CxEnS9PgQyv0W0xA7PE3LGNMzmx8" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="784" data-original-width="1352" height="371" src="https://blogger.googleusercontent.com/img/a/AVvXsEi3JcuuAO0hezR91BjbWiv4SJ6mtCuVYQtIHXFS9b2smnEtMoVPYbGL4W5wIpq8geopeib6cAQNT2_tkykuovlEsc_FLQzyf_hWZ3GqHLasC50pMkQB-fDuPpELGEZPQtzkVqDRUPWtW8qCzH6Yw4PYCk7CxEnS9PgQyv0W0xA7PE3LGNMzmx8=w640-h371" width="640" /></a></div><br /><p></p><div>Many many years back in an interview I was asked to design a solver for the game Wordament. At that time I had no idea what the game was and the interviewer patiently explained it to me. I later learnt that couple of engineers in Microsoft came up with the game for the Windows phone platform and it was such a success that they went and bootstrapped a team and made that game their full time job. </div><div><br /></div><div>I was able to give a solution in the interview, but that always remained at the back of my mind. I wanted to go further than the theoretical solution and really build the solver. I began tinkering with the idea a couple of weeks back and over the Thanksgiving long weekend I got enough time to sit down and complete the solution.</div><div><br /></div><div>The sources are at <a href="http://github.com/abhinababasu/wordament/">github.com/abhinababasu/wordament/</a></div><div>You can see it in action at <a href="http://bonggeek.com/wordament/">bonggeek.com/wordament/</a></div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjydrWGgFcLyi2KDcJ2lDour0MG_IPJ0lEvnJwTVcXCWsCPAK-YQBMPPMU11PqYebNMdT4GLWCVAkuzzSEfWSj2Qb88ymilXEpPWUa2M7h_dqjoTsjcDAd1EVPlvm-9BbHSRedAoqK-pnoWajPQMN88ngiJwdHJzid2xDw7zJX78sjNcJ78FMQ/s1000/chrome-capture-2022-10-25.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="899" data-original-width="1000" height="575" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjydrWGgFcLyi2KDcJ2lDour0MG_IPJ0lEvnJwTVcXCWsCPAK-YQBMPPMU11PqYebNMdT4GLWCVAkuzzSEfWSj2Qb88ymilXEpPWUa2M7h_dqjoTsjcDAd1EVPlvm-9BbHSRedAoqK-pnoWajPQMN88ngiJwdHJzid2xDw7zJX78sjNcJ78FMQ/w640-h575/chrome-capture-2022-10-25.gif" title="Wordament Screen recroding" width="640" /></a></div><br /><h2 style="text-align: left;">Basic Idea</h2><div>We begin by loading dictionary into a <a href="https://en.wikipedia.org/wiki/Trie">Trie </a>data-structure. Obviously there are fantastic Trie implementation out there, including ones that are highly optimized in memory by being able to collapse multiple nodes into one, however, the whole idea of this exercise was to write some code. So I rolled out a basic Trie.</div><div><br /></div><div>If a particular Trie node is a end of word, then that node is marked as so. As an example a Trie created with the words, cat, car, men, man, mad will look as below. The green checks denote these are valid end of word nodes.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwrnX_sOkOrtY-_ebrsXKSuwITQQ9yYrsUBKb84wmuZ3qSaU63YXjUD6UXVnnwNxZawQJUglSOTWeSK9B8Ien_fOi5klMUeT0VCJlvewsVrZ1PSaVX8qnMDcKZBtqKo7unOy4EhDCskrmvpmBTrRRqtJ9dyMGghzuheDk5LJ96njMkDMdlUu8/s916/Screenshot%202022-11-25%20213904.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="648" data-original-width="916" height="283" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwrnX_sOkOrtY-_ebrsXKSuwITQQ9yYrsUBKb84wmuZ3qSaU63YXjUD6UXVnnwNxZawQJUglSOTWeSK9B8Ien_fOi5klMUeT0VCJlvewsVrZ1PSaVX8qnMDcKZBtqKo7unOy4EhDCskrmvpmBTrRRqtJ9dyMGghzuheDk5LJ96njMkDMdlUu8/w400-h283/Screenshot%202022-11-25%20213904.png" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div>Now starting from each cell of Wordament, we start at the node for that cell character in the Trie. We look at the 8 adjacent cells (neighbors) and if there are Trie children with the same character as the neighbor, then it is a candidate to look into. And we recursively move to that node. At any point if we arrive at a valid word node, then we check to see if that word was previously found, if not, we add the word and the list of cells that created that word in the result.<div><br /></div><div>Finally since Wordament gives higher score for longer words, we sort the list of words by their length.</div><div><br /></div><div>The logic of this solution is implemented in <a href="https://github.com/abhinababasu/wordament/blob/main/solver/wordament.go">wordament.go</a>.</div><div><br /></div><div>I built the solver into a web-service, that runs in a docker container inside Azure VM. The service exposes an API. Then I built a single page web-application, that calls this web-service and renders the solution.</div><div><br /></div><div>You can hit the API directly with something like</div><div><span style="font-family: courier; font-size: x-small;">curl -s commonvm1.westus2.cloudapp.azure.com:8090/?input=SPAVURNYGERSMSBE | jq .</span></div><div><br /></div><div>The input is all the 16 characters of the Wordament to be solved.</div>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-81979063142109991462022-03-02T17:43:00.000-08:002022-03-02T18:49:30.664-08:00CAYL - Code as you like day<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Pv27uQVwSmSy2jTGJoWQmK1l0tNnxkz7NGMInACSUR-YeuZFhgX_422aluc9BT4vGs79CKCaRDKVSExRJImm8Yfas1pPMscTDHhk743Xl5sROqDDcahKqKcf3JSXLJPaEjZvug/s1600/IMG_3593-Edit.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1067" data-original-width="1600" height="425" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Pv27uQVwSmSy2jTGJoWQmK1l0tNnxkz7NGMInACSUR-YeuZFhgX_422aluc9BT4vGs79CKCaRDKVSExRJImm8Yfas1pPMscTDHhk743Xl5sROqDDcahKqKcf3JSXLJPaEjZvug/s640/IMG_3593-Edit.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Building an enterprise grade distributed service is like trying to fix and improve a car while driving it at high speed down the free-way. Engineering debt accumulates fast and engineers in the team yearn for the time to get to them. A common complaint is also that we need more time to tinker with cool features and tech to learn and experiment.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
An approach many companies take is the big hackathon events. Even though they have their place, I think those are mostly for PR and getting eye candy. Which exec doesn’t want to show the world their company creates AI powered blockchain running on quantum computer in just a 3 day hackathon.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div style="clear: both; text-align: left;">
This is where CAYL comes in. CAYL or “Code As You Like” is named loosely on “go as you like” event I experienced as a student in India. In a lot of uniform based schools in Kolkata, it is common to have a go as you like day, where kids dress up however they want.</div>
<div style="clear: both; text-align: left;">
<br /></div>
<div style="clear: both; text-align: left;">
Even though we call it code as you like, it has evolved beyond coding. One of our extended Program Management team has also picked this up and call it the WAYL (Work as you like day). This is what we have set aside in our group calendar for this event.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<span style="font-family: "courier new";">“code as you like day” is a reserved date every month (first Monday of the month) where we get to code/document or learn something on our own.</span><br />
<span style="font-family: "courier new";"> There will be no scheduled work items and no standups.</span><br />
<span style="font-family: "courier new";">We simply do stuff we want to do. Examples include but not limited to</span><br />
<ol>
<li><span style="font-family: "courier new";">Solve a pet peeve (e.g. fix a bug that is not scheduled but you really want to get done)</span></li>
<li><span style="font-family: "courier new";">A cool feature</span></li>
<li><span style="font-family: "courier new";">Learn something related to the project that you always wanted to figure out (how do we use fluentd to process events, what is helm)</span></li>
<li><span style="font-family: "courier new";">Learn something technical (how does go channels work, go assembly code)</span></li>
<li><span style="font-family: "courier new";">Shadow someone from a sibling team and learn what they are working on</span></li>
</ol>
<span style="font-family: "courier new";">We can stay late and get things done (you totally do not have to do that) and there will be pizza or ice-cream. </span><br />
<div class="separator" style="clear: both;">
<span style="font-family: "courier new";"></span><span style="font-family: "courier new";"></span></div>
<span style="font-family: "courier new";">One requirement is that you *have* to present the next day, whatever you did. 5 minutes each</span><br />
<br />
I would say we have had great success with it. We have had CAYL projects all over the spectrum<br />
<ol>
<li>Speed up build system and just make building easier</li>
<li>ML Vision device that can tell you which bin trash needs to go in (e.g. if it is compostable)</li>
<li>Better BVT system and cross porting it to work on our Macs</li>
<li>Pet peeves like make function naming more uniform, remove TODO from code, spelling/grammar etc.</li>
<li>Better logging and error handling</li>
<li>Fix SQL resiliency issues</li>
<li>Move some of our older custom management VMs move to AKS</li>
<li>Bring in gomock, go vet, static checking</li>
<li>3D game where mommy penguin gets fish for her babies and learns to be optimal using machine learning</li>
<li>Experiment with Prometheus </li>
<li>A dev spent a day shadowing dev from another team to learn the cool tech they are using etc.</li>
</ol>
We just finished our CAYL yesterday and one of my CAYL items was to write a blog about it. So it’s fitting that I am hitting publish on this blog, as I sit in the CAYL presentation while eating Kale chips</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-78381809495237833742022-02-07T00:43:00.005-08:002022-02-07T00:48:46.334-08:00Go Generics<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgo_vCT2CuE2dnLPIyN1FOJGMIfQzg-fgevH_fbrD-WvqunRlKSo2bfsiM96whKe3aniU42buRqfq6IZU11uXURlmx9rq01CEcXJnJ8sf8ZVaxz34lsMvYVq-j0nCPBu11RA6-JpfMu4gNI9el_yWYsjpgbjflGsD2038BtoPxD8ad_qnIb76o=s959" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="627" data-original-width="959" height="418" src="https://blogger.googleusercontent.com/img/a/AVvXsEgo_vCT2CuE2dnLPIyN1FOJGMIfQzg-fgevH_fbrD-WvqunRlKSo2bfsiM96whKe3aniU42buRqfq6IZU11uXURlmx9rq01CEcXJnJ8sf8ZVaxz34lsMvYVq-j0nCPBu11RA6-JpfMu4gNI9el_yWYsjpgbjflGsD2038BtoPxD8ad_qnIb76o=w640-h418" width="640" /></a></div><br /><p>Every month in our team we do a <a href="https://blog.bonggeek.com/2019/10/cayl-code-as-you-like-day.html">Code as You Like Day</a>, which is basically a day of taking time off regular work and hacking something up, learning something new or even fixing some pet-peeves in the system. This month I chose to learn about go-lang generics.</p><p>I started go many years back while coming from mainly coding in C++ and C#. Also in Adobe almost 20 years back I got a week long class on generic programming from <a href="https://en.wikipedia.org/wiki/Alexander_Stepanov">Alexander Stepanov</a> himself. I missed generics terribly and hated all the code I had hand role out for custom container types. So I was looking forward to generics in go.</p><p>This was also the first time I was trying to use a non-stable version of go as generics is available currently as go 1.18 Beta 2. Installing this was a bit confusing for me.</p><p>I just attempted go install which seemed to work</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgkMJQO5j337w7RtXCwIbccr5csV0JBJL8P_PMZLi8hLUEdVSV71y5jMyTpv9KiDdRuF9gqu2JHzGLHTSz0SJBMbxckkq3Z7Xibrz5CQt9f7wk9h4ak_k0pk1tsebVaCH8-8EalyGkN8YM9RJRJ-8u9aQqQ0D-QBeSfZAPioTbPTrfaIN-z_aQ=s527" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="78" data-original-width="527" height="94" src="https://blogger.googleusercontent.com/img/a/AVvXsEgkMJQO5j337w7RtXCwIbccr5csV0JBJL8P_PMZLi8hLUEdVSV71y5jMyTpv9KiDdRuF9gqu2JHzGLHTSz0SJBMbxckkq3Z7Xibrz5CQt9f7wk9h4ak_k0pk1tsebVaCH8-8EalyGkN8YM9RJRJ-8u9aQqQ0D-QBeSfZAPioTbPTrfaIN-z_aQ=w640-h94" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiglB-LDC9QPEYt3aq_lrLXP47wKwAR_Gtatw2V3kKat9fgo4GgRiagM5A9K7JynCNTpWJ42iQG6eQkoBrFXgdxcVstHlO7QIDdQtG53crnQxPXLfbosxV89jwSoSxW8UX-1IqhtSGnlhV9Ozwz-qE_HmUw71lSxpzF7zXOwD0tbuJOlmq3kF0=s840" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiglB-LDC9QPEYt3aq_lrLXP47wKwAR_Gtatw2V3kKat9fgo4GgRiagM5A9K7JynCNTpWJ42iQG6eQkoBrFXgdxcVstHlO7QIDdQtG53crnQxPXLfbosxV89jwSoSxW8UX-1IqhtSGnlhV9Ozwz-qE_HmUw71lSxpzF7zXOwD0tbuJOlmq3kF0=s840" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiglB-LDC9QPEYt3aq_lrLXP47wKwAR_Gtatw2V3kKat9fgo4GgRiagM5A9K7JynCNTpWJ42iQG6eQkoBrFXgdxcVstHlO7QIDdQtG53crnQxPXLfbosxV89jwSoSxW8UX-1IqhtSGnlhV9Ozwz-qE_HmUw71lSxpzF7zXOwD0tbuJOlmq3kF0=s840" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="248" data-original-width="840" height="189" src="https://blogger.googleusercontent.com/img/a/AVvXsEiglB-LDC9QPEYt3aq_lrLXP47wKwAR_Gtatw2V3kKat9fgo4GgRiagM5A9K7JynCNTpWJ42iQG6eQkoBrFXgdxcVstHlO7QIDdQtG53crnQxPXLfbosxV89jwSoSxW8UX-1IqhtSGnlhV9Ozwz-qE_HmUw71lSxpzF7zXOwD0tbuJOlmq3kF0=w640-h189" width="640" /></a></div><div>but seemed like it did not work. I had to do an additional step of download. That wasn't very intuitive.</div><div><br /><div class="separator" style="clear: both; text-align: left;">For my quick test, I decided to do a port my quick and dirty stack implementation from relying on interface{} to use generic type.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">I created a Stack with generic type T which is implemented over a slice of T.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">Full</span> = errors.<span style="color: #dcdcaa;">New</span>(<span style="color: #ce9178;">"Full"</span>)</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">Empty</span> = errors.<span style="color: #dcdcaa;">New</span>(<span style="color: #ce9178;">"Empty"</span>)</div><br /><div><span style="color: #569cd6;">type</span> <span style="color: #4ec9b0;">Stack</span>[<span style="color: #9cdcfe;">T</span> any] <span style="color: #569cd6;">struct</span> {</div><div> <span style="color: #9cdcfe;">arr</span> []<span style="color: #4ec9b0;">T</span></div><div> <span style="color: #9cdcfe;">curr</span> <span style="color: #4ec9b0;">int</span></div><div> <span style="color: #9cdcfe;">max</span> <span style="color: #4ec9b0;">int</span></div><div>}</div></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Creating two functions to create a fixed size stack or growable was a breeze. Using the generic types was intuitive.</div><div class="separator" style="clear: both; text-align: left;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><br /><div><span style="color: #569cd6;">func</span> <span style="color: #dcdcaa;">NewSizedStack</span>[<span style="color: #9cdcfe;">T</span> any] (<span style="color: #9cdcfe;">size</span> <span style="color: #4ec9b0;">int</span>) *<span style="color: #9cdcfe;">Stack</span>[T] {</div><div> <span style="color: #9cdcfe;">s</span> := &<span style="color: #9cdcfe;">Stack</span>[T]{max: size}</div><br /><div> <span style="color: #9cdcfe;">s.arr</span> = <span style="color: #dcdcaa;">make</span>([]<span style="color: #4ec9b0;">T</span>, <span style="color: #9cdcfe;">size</span>)</div><div> <span style="color: #c586c0;">return</span> s</div><div>}</div><br /><div><span style="color: #569cd6;">func</span> <span style="color: #dcdcaa;">NewStack</span>[<span style="color: #9cdcfe;">T</span> any]() *<span style="color: #9cdcfe;">Stack</span>[T] {</div><div> <span style="color: #c586c0;">return</span> &<span style="color: #9cdcfe;">Stack</span>[T]{</div><div> max: math.MaxInt32,</div><div> }</div><div>}</div><br /></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">However, I did fumble on creating the methods on that type. Because I somehow felt I need to write it as func (s *Stack[T])Length[T any]() int {}. However, the [T any] is actually not required.</div><div class="separator" style="clear: both; text-align: left;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><br /><div><span style="color: #569cd6;">func</span> (<span style="color: #9cdcfe;">s</span> *<span style="color: #9cdcfe;">Stack</span>[T]) <span style="color: #dcdcaa;">Length</span>() <span style="color: #4ec9b0;">int</span> {</div><div> <span style="color: #c586c0;">return</span> s.curr</div><div>}</div><br /><div><span style="color: #569cd6;">func</span> (<span style="color: #9cdcfe;">s</span> *<span style="color: #9cdcfe;">Stack</span>[T]) <span style="color: #dcdcaa;">IsEmpty</span>() <span style="color: #4ec9b0;">bool</span> {</div><div> <span style="color: #c586c0;">return</span> s.<span style="color: #dcdcaa;">Length</span>() == <span style="color: #b5cea8;">0</span></div><div>}</div><br /></div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Push and Pop worked out as well</div><div class="separator" style="clear: both; text-align: left;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><br /><div><span style="color: #569cd6;">func</span> (<span style="color: #9cdcfe;">s</span> *<span style="color: #9cdcfe;">Stack</span>[T]) <span style="color: #dcdcaa;">Push</span>(<span style="color: #9cdcfe;">v</span> T) <span style="color: #4ec9b0;">error</span> {</div><div> <span style="color: #c586c0;">if</span> s.<span style="color: #9cdcfe;">curr</span> == <span style="color: #dcdcaa;">len</span>(s.<span style="color: #9cdcfe;">arr</span>) {</div><div> <span style="color: #c586c0;">if</span> s.<span style="color: #9cdcfe;">curr</span> == s.max {</div><div> <span style="color: #c586c0;">return</span> Full</div><div> } <span style="color: #c586c0;">else</span> {</div><div> <span style="color: #9cdcfe;">s.arr</span> = <span style="color: #dcdcaa;">append</span>(s.<span style="color: #9cdcfe;">arr</span>, <span style="color: #9cdcfe;">v</span>)</div><div> }</div><div> } <span style="color: #c586c0;">else</span> {</div><div> s.<span style="color: #9cdcfe;">arr</span>[s.curr] = v</div><div> }</div><br /><div> s.<span style="color: #9cdcfe;">curr</span>++</div><br /><div> <span style="color: #c586c0;">return</span> <span style="color: #569cd6;">nil</span></div><div>}</div><br /><div><span style="color: #569cd6;">func</span> (<span style="color: #9cdcfe;">s</span> *<span style="color: #9cdcfe;">Stack</span>[T]) <span style="color: #dcdcaa;">Pop</span>() (T, <span style="color: #4ec9b0;">error</span>) {</div><div> <span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">noop</span> <span style="color: #4ec9b0;">T</span> <span style="color: #6a9955;">// 0 value</span></div><div> <span style="color: #c586c0;">if</span> s.<span style="color: #dcdcaa;">Length</span>() == <span style="color: #b5cea8;">0</span> {</div><div> <span style="color: #c586c0;">return</span> noop, Empty</div><div> }</div><br /><div> <span style="color: #9cdcfe;">v</span> := s.<span style="color: #9cdcfe;">arr</span>[s.curr-<span style="color: #b5cea8;">1</span>]</div><div> s.<span style="color: #9cdcfe;">arr</span>[s.curr-<span style="color: #b5cea8;">1</span>] = noop <span style="color: #6a9955;">// release the reference</span></div><div> s.<span style="color: #9cdcfe;">curr</span>--</div><br /><div> <span style="color: #c586c0;">return</span> v, <span style="color: #569cd6;">nil</span></div><div>}</div></div></div><div class="separator" style="clear: both; text-align: left;">However, for pop I needed to return a nil/0-value for the generic type. It did seem odd that go does not implement something specific for it. I had to create a variable as noop and they return that.</div></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Using the generic type is a breeze too, no more type casting!</div><div class="separator" style="clear: both; text-align: left;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #9cdcfe;">s</span> := <span style="color: #9cdcfe;">NewStack</span>[<span style="color: #4ec9b0;">int</span>]()</div><div><br /></div><div>s.<span style="color: #dcdcaa;">Push</span>(<span style="color: #b5cea8;">5</span>)</div><div style="line-height: 19px;"><div><span style="color: #c586c0;">if</span> <span style="color: #9cdcfe;">v</span>, <span style="color: #9cdcfe;">e</span> := s.<span style="color: #dcdcaa;">Pop</span>(); e != <span style="color: #569cd6;">nil</span> {</div><div> t.<span style="color: #dcdcaa;">Errorf</span>(<span style="color: #ce9178;">"Should get poped value"</span>)</div><div>}</div></div></div></div>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-15269385157558431242020-06-16T07:42:00.000-07:002020-06-16T07:42:00.183-07:00Raspberry Pi Photo frame<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-IWXe4IBD3O6ZAOAaOpBTmLHKoTJVB2p77LUPUK5HHbwkMFAHWBxfEYttA77fIUMtoYy_gKXhXvuQ9lustuLMhY1sy9tVIISlze_-49a2FircbxD4qavNCJ1TxFfcOi0ulMZfyw/s1600/IMG_5623-Pano.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1203" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-IWXe4IBD3O6ZAOAaOpBTmLHKoTJVB2p77LUPUK5HHbwkMFAHWBxfEYttA77fIUMtoYy_gKXhXvuQ9lustuLMhY1sy9tVIISlze_-49a2FircbxD4qavNCJ1TxFfcOi0ulMZfyw/s640/IMG_5623-Pano.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
This small project brings together bunch of my hobbies together. I got to play with carpentry, photography and software/technology including face detection.<br />
<br />
I have run out of places in the home to hang photo frames and as a way around I was planning to get a digital photo frame. When I upgraded my home desktop to 2 x 4K monitors I had my old dell 28" 1080p monitor lying around. I used that and a raspberry pi to create a photo frame. It boasts of the following features<br />
<ol style="text-align: left;">
<li>A real handmade frame</li>
<li>1080p display</li>
<li>Auto sync from OneDrive</li>
<li>Remotely managed</li><li>Face detection based image crop</li><li>Low cost (uses raspberry pi)</li>
</ol>
<div>
This is how it looks.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw2WrBhqlcGGihwbRxabON3_6yK30O0-R2oimQyXUhhbpC9ZMWaR61o5URFGyHQEwjHbP1k7t-eYtmMHV-0FjDkReocuScD88Cz5ko2R9yFRDPmKP2VHwRWGhvkBzSmtQHX5l1Hg/s2798/20200616_024041.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1960" data-original-width="2798" height="448" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw2WrBhqlcGGihwbRxabON3_6yK30O0-R2oimQyXUhhbpC9ZMWaR61o5URFGyHQEwjHbP1k7t-eYtmMHV-0FjDkReocuScD88Cz5ko2R9yFRDPmKP2VHwRWGhvkBzSmtQHX5l1Hg/w640-h448/20200616_024041.jpg" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div></div>
<div>
<h2 style="text-align: left;">
Construction</h2>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoda9gLlGZpMBRXyyf-WR1CTS5XhiCC9UI7bu13ezoEysITwg-ONRkmywcZpBBNJjRdaOoXwDjuDiYKyci9HGEMqqSq6Jlfa7pyFxlWS9X1MBSDS9abLzJyHaIn8mfoakZ4LFtmw/s1600/20200611_182636.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1600" data-original-width="1136" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoda9gLlGZpMBRXyyf-WR1CTS5XhiCC9UI7bu13ezoEysITwg-ONRkmywcZpBBNJjRdaOoXwDjuDiYKyci9HGEMqqSq6Jlfa7pyFxlWS9X1MBSDS9abLzJyHaIn8mfoakZ4LFtmw/s200/20200611_182636.jpg" width="141" /></a>In my previous project of <a href="https://www.youtube.com/watch?v=mCrHooiS1vM">smart-mirror</a>, I focused way too much on the framing monitor part and finally had the problem that the raspberry-pi and the monitor is so well contained inside the frame that I have a hard time accessing it and replacing stuff. So this time my plan was to build a simple lightweight frame that is put on the monitor using velcro fasteners so that I can easily remove the frame. The monitor is actually on its own base, so the frame is just cosmetic and doesn't bear the load of the monitor. Rather the monitor and its base holds the frame in place.</div>
<div>
<br /></div>
<div>
I bought a 2" trim from Homedepot and cut out 4 pieces using a saw and then joined them using just wood glue. To let the glue cure, I held the corners using corner clamp for 12 hours. The glue is actually stronger than the trim itself, so once it dries there is no chance of things falling apart.</div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6L4YEdUinHV0Db-l821KNbbhtlJpCw90vgt4KtpeI1QwzLxhWuPwzzm8Hb3RUkz1IRxFilF2yRECpiAgYYh9gPKYDduH16Til6_BOOTIJtF6N_Ni2W8Vw2XaEPIXJB4DspHVUw/s1600/20200611_183754.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1600" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6L4YEdUinHV0Db-l821KNbbhtlJpCw90vgt4KtpeI1QwzLxhWuPwzzm8Hb3RUkz1IRxFilF2yRECpiAgYYh9gPKYDduH16Til6_BOOTIJtF6N_Ni2W8Vw2XaEPIXJB4DspHVUw/s640/20200611_183754.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyM5j3Fga-skuINd5olZsH094I-tVIRE7QjrnxCWTd7LxdCbt-ZanvpLtZRYlE4h9jUv-1zWpjdObDCXk1ax8PgB7r2TH0BmHCYaFoRL6FmTOl81HxSx0NPMz48dwEdf3xvk-zVA/s1600/20200611_192219.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="956" data-original-width="1600" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyM5j3Fga-skuINd5olZsH094I-tVIRE7QjrnxCWTd7LxdCbt-ZanvpLtZRYlE4h9jUv-1zWpjdObDCXk1ax8PgB7r2TH0BmHCYaFoRL6FmTOl81HxSx0NPMz48dwEdf3xvk-zVA/s640/20200611_192219.jpg" width="640" /></a></div>
<br />
On the back of the frame I attached a small piece of wood, on which I added velcro. I also glued velcro to the top of the monitor. These two strips of velcro keeps the frame on the monitor.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUpOrfU8jBz5LaPgqaybSc1seiVwYz2F09yl_kGkINA_D7WvNMqmPDVxAcb0On2VymNfQlRwLajv-0ckPXl70ntXQnjIR4Fu8QqG9QrPTCOuSKcS8O2_y8yy5NuIxo1lDTgHR3YQ/s1600/20200614_012924.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1600" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUpOrfU8jBz5LaPgqaybSc1seiVwYz2F09yl_kGkINA_D7WvNMqmPDVxAcb0On2VymNfQlRwLajv-0ckPXl70ntXQnjIR4Fu8QqG9QrPTCOuSKcS8O2_y8yy5NuIxo1lDTgHR3YQ/s640/20200614_012924.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFZ8qCHVKBBtXnmVQz9F6aD3kN9VnjftiVDEMGiBYmCjDdPMaFW5QrAUv8d31CkEtV_HFp_lfcd6fUnshB8-uLNUgz-of9T6kbcrjiszqAMhtfAMtOsBlDoxxsymvowzhVaU8ZNw/s1600/20200614_012933.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1099" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFZ8qCHVKBBtXnmVQz9F6aD3kN9VnjftiVDEMGiBYmCjDdPMaFW5QrAUv8d31CkEtV_HFp_lfcd6fUnshB8-uLNUgz-of9T6kbcrjiszqAMhtfAMtOsBlDoxxsymvowzhVaU8ZNw/s640/20200614_012933.jpg" width="438" /></a></div>
<br />
Now the frame can be attached loosely to the monitor just by placing on it.<br />
<br />
After that I got a raspberry-pi and connected it to the monitor using hdmi cable and attached the raspberry pi with zip ties to the frame. All low tech till this point.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibNqTmWPL2v-SHp1ZxWhDtX25QOody2izJHyxTULYBQKqUPqLQJA72yBQD9B0yV8CfAyIRaF05RTpZp5TXFMnvtDUPEkp-xAw_hQwSPlWd6CKMnf__NIob6Ut4SwTsmxDRi6SWsw/s1600/20200614_014822.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="778" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibNqTmWPL2v-SHp1ZxWhDtX25QOody2izJHyxTULYBQKqUPqLQJA72yBQD9B0yV8CfAyIRaF05RTpZp5TXFMnvtDUPEkp-xAw_hQwSPlWd6CKMnf__NIob6Ut4SwTsmxDRi6SWsw/s640/20200614_014822.jpg" width="310" /></a></div>
On powering up, it boots into Raspbian.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-881kJfpXrXq8SMX7SKzGVnoxr5XfRpX3n-5_cuxMMRn5I_LgVMM0LFP19b0XoQ4NJYx3RnrKHgFCAhvu3O0mgg7QO6kVihmiCl7IBetMSz8hIpHnbvBSPwYK5E3Q4vu52phd8w/s1600/20200614_011804.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="941" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-881kJfpXrXq8SMX7SKzGVnoxr5XfRpX3n-5_cuxMMRn5I_LgVMM0LFP19b0XoQ4NJYx3RnrKHgFCAhvu3O0mgg7QO6kVihmiCl7IBetMSz8hIpHnbvBSPwYK5E3Q4vu52phd8w/s640/20200614_011804.jpg" width="376" /></a></div>
<br />
<h2 style="text-align: left;">
Software</h2>
<div>
<h3 style="text-align: left;">
Base Setup</h3>
<div>
I always get my base setup </div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get update</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get upgrade</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get install xrdp # install remote desktop</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get install vim # my editor of choice</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get install git</span></div>
<div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new", courier, monospace;">git clone https://github.com/abhinababasu/share # get my shell</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">cp share/.vimrc .</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">cp share/.bash_aliases .</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">cp share/.bashrc .</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;">cp share/.bash_aliases .</span></div>
<div>
<span style="font-family: "courier new", courier, monospace;"><br /></span></div>
</div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo apt-get install unclutter # hide mouse pointer in slide show</span></div>
<div>
<br /></div>
<div>
To keep things fresh, reboot midnight every day, add the following to /etc/crontab</div>
<div>
<span style="font-family: "courier new", courier, monospace;">0 0 * * * root reboot</span></div>
<div>
<br /></div>
<div>
Enable ssh</div>
<div>
<span style="font-family: "courier new", courier, monospace;">sudo raspi-config</span></div>
<div>
<br /></div>
<div>
Portrait mode</div>
<div>
<span style="white-space: pre;"> </span>1. <span style="font-family: "courier new", courier, monospace;">sudo vim /boot/config.txt</span></div>
<div>
<span style="white-space: pre;"> </span>2. Add the line: <span style="font-family: "courier new", courier, monospace;">display_rotate=3</span></div>
<div>
<br /></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<h3 style="text-align: left;">
Push Pics</h3>
I use FrameMaker for managing photos I take. My workflow for this case is as follows<br />
<br />
<ol style="text-align: left;">
<li>All images are tagged with keyword "frame" in lightroom. </li>
<li>I use smart folder to see all these images and then publish to a folder named Frame in OneDrive</li>
</ol>
<br />
<h3 style="text-align: left;">
Sync OneDrive to Raspberry Pi</h3>
<div>
<div>
I used the steps in https://jarrodstech.net/how-to-raspberry-pi-onedrive-sync/ </div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: "courier new", courier, monospace;">curl -L https://raw.github.com/pageauc/rclone4pi/master/rclone-install.sh | bash</span></li>
<li><span style="font-family: "courier new", courier, monospace;">rclone config</span></li>
<ol>
<li>Enter n (for a new connection) and then press enter</li>
<li>Enter a name for the connection (i’ll enter onedrive) and press enter</li>
<li>Enter the number for One Drive</li>
<li>Press Enter for client ID</li>
<li>Press Enter for Client Secret</li>
<li>Press n and enter for edit advanced config</li>
<li>Enter y for auto config</li>
<li>A browser window will now open, log in with your Microsoft Account and select yes to allow OneDrive</li>
<li>Choose right option for OneDrive personal</li>
<li>Now select the OneDrive you would like to use, you will probably only have one OneDrive linked to your account. This will be 0</li>
<li>Y for subsequent questions</li>
</ol>
<li>To Sync once: <span style="font-family: "courier new", courier, monospace;">rclone sync -v onedrive:Frame /home/pi/frame</span></li>
<li>Setup automatic sync every one hour</li>
<ol>
<li><span style="font-family: "courier new", courier, monospace;">echo "rclone sync -v onedrive:Frame /home/pi/frame" > ~/sync.sh</span></li>
<li><span style="font-family: "courier new", courier, monospace;">chmod +x ~/sync.sh</span></li>
<li><span style="font-family: "courier new", courier, monospace;">crontab -e</span></li>
<li>Add the line: <span style="font-family: "courier new", courier, monospace;">1 * * * * /home/pi/sync.sh</span></li>
</ol>
</ol>
</div>
<h3 style="text-align: left;">
Setup Screensaver</h3>
</div>
<div>
There are many options that I could find online to show the photos. But I chose to go with the easiest one, use the xscreensaver. However, there are some issues and most likely this is something I will revisit.</div>
<div>
<br /></div>
<div>
<ol style="font-size: 11pt; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; text-align: left; unicode-bidi: embed;" type="1">
<li style="font-family: calibri;"><span style="font-family: calibri; font-size: 11pt;">Disable screen blanking after some time of no use</span></li>
<ol style="font-size: 11pt; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; unicode-bidi: embed;" type="a">
<li><span style="font-size: 11pt;"><span style="font-family: "courier new", courier, monospace;">vi
/etc/lightdm/lightdm.conf</span></span></li>
<li><span style="font-family: calibri; font-size: 11pt;">Addd the line</span><span style="font-family: "courier new", courier, monospace;">[SeatDefaults]<br />xserver-command=X
-s 0 -dpms<br /><br /></span></li>
</ol>
<li style="font-family: calibri;">Enable auto-login, so that on restart you directly get logged in and then into screensaver</li>
<ol>
<li><span style="font-family: "courier new", courier, monospace;">sudo
raspi-config</span></li>
<li style="font-family: calibri;"><span style="font-family: calibri; font-size: 11pt;">Select
'Boot Options' then 'Desktop / CLI' then 'Desktop Autologin'. Then right arrow
twice and Finish and reboot.<br /><br /></span></li>
</ol>
<li style="font-family: calibri;"><span style="font-family: calibri; font-size: 11pt;"> </span>Setup screen saver</li>
<ol style="font-size: 11pt; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; unicode-bidi: embed;" type="a">
<li><span style="font-size: 11pt;"><span style="font-family: "courier new", courier, monospace;">sudo apt-get -y
install xscreensaver</span></span></li>
<li><span style="font-size: 11pt;"><span style="font-family: "courier new", courier, monospace;">sudo apt-get -y install
xscreensaver-gl-extra</span></span></li>
</ol>
</ol>
</div>
<br />
These are my screen saver settings to show the photos in /home/pi/frame as slideshow<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyuvqZXYykiDk5dHb_RJm1IosfDlHxD3gAg3Z3GUR0jzF4GKHpf-Ol7sMuX7AVLZbPxbMuJ3T9p5itR4XNxSDj8mgRkShxZVj7IwpM2iPIvMUhWRFSW1o5pE8BOfFF4wX725sGOQ/s1600/0.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="572" data-original-width="524" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyuvqZXYykiDk5dHb_RJm1IosfDlHxD3gAg3Z3GUR0jzF4GKHpf-Ol7sMuX7AVLZbPxbMuJ3T9p5itR4XNxSDj8mgRkShxZVj7IwpM2iPIvMUhWRFSW1o5pE8BOfFF4wX725sGOQ/s400/0.PNG" width="366" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5-AOKK0ZSBdOeZN4WY9uRQFCdDbDy9PnCdOxz6m3FIH_Za-f1aR1Ujt2DLcb9aiq65IsMMtoG9A37JWdyYCMCtpdS8rstc1LAOqFVyFvM0KxVxidYbFwJ3_RR9iBdFKsXE8N2YQ/s1600/1.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="652" data-original-width="834" height="500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5-AOKK0ZSBdOeZN4WY9uRQFCdDbDy9PnCdOxz6m3FIH_Za-f1aR1Ujt2DLcb9aiq65IsMMtoG9A37JWdyYCMCtpdS8rstc1LAOqFVyFvM0KxVxidYbFwJ3_RR9iBdFKsXE8N2YQ/s640/1.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf_ilBL20u5KSdSz1M0HJal5jFvGxsUMX78OA_bBPDSPz0u7bow97DARYxnPYTPwMY01AcYWXb5ALI8oVuX1bpQJe6rdYsyXfgVpR5lp7ZS2fxNl4OSI4k1JxWXDA5MA81OkbWBQ/s1600/2.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="636" data-original-width="814" height="500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf_ilBL20u5KSdSz1M0HJal5jFvGxsUMX78OA_bBPDSPz0u7bow97DARYxnPYTPwMY01AcYWXb5ALI8oVuX1bpQJe6rdYsyXfgVpR5lp7ZS2fxNl4OSI4k1JxWXDA5MA81OkbWBQ/s640/2.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCa4AT3teJc15bFj-tk1tDiBGQX8AC2PmuRLdR51q-zv3dOkT-ZwLO1KCW4EN_wDjMzoTDobKCrrA3JDNps4_youj_SK3bgcFTYAM2ZBtwQKQGbFiBlQAtxW8ZmBN3pYE0mYbTg/s1600/3.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="635" data-original-width="810" height="500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCa4AT3teJc15bFj-tk1tDiBGQX8AC2PmuRLdR51q-zv3dOkT-ZwLO1KCW4EN_wDjMzoTDobKCrrA3JDNps4_youj_SK3bgcFTYAM2ZBtwQKQGbFiBlQAtxW8ZmBN3pYE0mYbTg/s640/3.png" width="640" /></a></div>
<br />
<h3 style="text-align: left;">Problems and solving with Face Detection</h3>
</div>
<div>My photos are rarely 9:16 portraits, that means an ugly black box on the top and bottom of the images. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNM4MujukeI2Ka_XjHpAZtJbcmaBHl9GPwUk46p0ZCPieWShiHPph8x05ciayIPAf_ym7RRVSctr15P1vROYzaq6luzkE4EP58Y2K0QvjFnROPE4BwGr5p27BxRAB-xJf-Owl7YQ/s2869/20200616_015343.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2869" data-original-width="1874" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNM4MujukeI2Ka_XjHpAZtJbcmaBHl9GPwUk46p0ZCPieWShiHPph8x05ciayIPAf_ym7RRVSctr15P1vROYzaq6luzkE4EP58Y2K0QvjFnROPE4BwGr5p27BxRAB-xJf-Owl7YQ/w418-h640/20200616_015343.jpg" width="418" /></a></div><div><br /></div><div>Obvious approach is to crop using some batch tool. But that would mean the crop could arbitrarily cut images out. Consider the following image </div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2j9FE1NqVEU7S_TC0fNDig0euGvHPrFxc_KaB5yeBLLS9iExooqgt20a4egDOTr2hB0k3PPtaXez6aBcknliCNFKXwZaghM5PqAr5mSZmav4aUgwwBdXpHAyEjgU3k5SmA7Imug/s2160/IMG_1688.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1168" data-original-width="2160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2j9FE1NqVEU7S_TC0fNDig0euGvHPrFxc_KaB5yeBLLS9iExooqgt20a4egDOTr2hB0k3PPtaXez6aBcknliCNFKXwZaghM5PqAr5mSZmav4aUgwwBdXpHAyEjgU3k5SmA7Imug/s320/IMG_1688.jpg" width="320" /></a></div><div>Cropping in a batch tool that picks up arbitrary area of the image generated something like below, which is obviously not acceptable.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2JZc-Qj3JWnNzT5-4jYEVuUbvuUuyZa6X639k5rUBtjedVBFOL7zY-0bvb9FDubUzp6FELaeGLHGABvl8mMdY0a3U_IILEgxEGMavfNOQCc6wkhMWZEE0v8Re1Mbmvka_lIiYCg/s1168/IMG_1688.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1168" data-original-width="657" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2JZc-Qj3JWnNzT5-4jYEVuUbvuUuyZa6X639k5rUBtjedVBFOL7zY-0bvb9FDubUzp6FELaeGLHGABvl8mMdY0a3U_IILEgxEGMavfNOQCc6wkhMWZEE0v8Re1Mbmvka_lIiYCg/s320/IMG_1688.jpg" /></a></div><div>To solve this I build a tool at <a href="https://github.com/abhinababasu/img">https://github.com/abhinababasu/img</a>. It takes my other project on detecting faces in images and then ensures that in the cropped image the face is retained. E.g. the tool above generates the following image.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUNVXsurKYTV5mbH5twDD4wuqB676TNx-oNJJfxQ3mjiKTCIyStJ1hX1AAL__8l9uXR2JE8YrAEjsI9pSzZ6ZmApCPVuapdMnpd6rw8CkScOXVmr_MwvcM9wPL1-bU6oFhdT6FMA/s1168/IMG_1688.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1168" data-original-width="657" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUNVXsurKYTV5mbH5twDD4wuqB676TNx-oNJJfxQ3mjiKTCIyStJ1hX1AAL__8l9uXR2JE8YrAEjsI9pSzZ6ZmApCPVuapdMnpd6rw8CkScOXVmr_MwvcM9wPL1-bU6oFhdT6FMA/s320/IMG_1688.jpg" /></a></div><div><br /></div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636695 -122.9775178 47.9487495 -121.68662379999999tag:blogger.com,1999:blog-6434675.post-22957523062538753162020-06-15T02:30:00.001-07:002020-07-07T13:52:19.279-07:00Building Azure Monitor for SAP Solutions <div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8TkCZVJYHGxGD7XqAIbrNmbMERhyvJlfXtFUnGO2bhpISOGN1XTde7fXz3YwaiHnlnq58mefZso6xEvNJO7_MS7JMke33vPrxs65VmtWBY6lsf5ZoRzvVCE-tRvWMDV9-4zTvqg/s1600/IMG_0417-Pano-Edit.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1238" data-original-width="1600" height="494" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8TkCZVJYHGxGD7XqAIbrNmbMERhyvJlfXtFUnGO2bhpISOGN1XTde7fXz3YwaiHnlnq58mefZso6xEvNJO7_MS7JMke33vPrxs65VmtWBY6lsf5ZoRzvVCE-tRvWMDV9-4zTvqg/s640/IMG_0417-Pano-Edit.jpg" width="640" /></a></div>
<div style="text-align: left;">
<h2 style="text-align: left;">
The Product</h2>
Update: Here is a quick-start video</div><div style="text-align: left;"> <iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/oWxtxAm-YRQ" width="480"></iframe> </div><div style="text-align: left;"><br /></div><div style="text-align: left;">This post is about how we <b>build</b> the Azure Monitor for Sap Solutions. It is about the distributed systems we use to build database monitoring at scale for customer's data-plane. However, the first section provides a quick intro into the product itself.<br />
<br />
"<i>Azure Monitor for SAP Solutions</i>" provides managed monitoring for the databases powering customer's SAP landscapes. Our monitoring supports multiple instances of databases of a particular type (e.g. HANA) and is also extendable for various kinds of databases. We have started with HANA and plan to include SQL-Server, etc. in the future. At the time of writing this post the monitoring is in private-preview with public preview coming up "soon".</div>
<div>
<br /></div>
<div>
The customer uses a creation wizard on Azure portal to create the monitor as shown is the screenshot below. Customer enters their subscription, resource group, vnet details, followed by connection details of the database. Our resource provider deploys a VM payload into their vnet that connects with the database to monitor them and pumps telemetry into their Azure analytics workspace. Customers can then create dashboards and configure alerts.</div>
<div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMoQ6txi5I1vl7LSSbRalsmqVw8vQC-QqLV_6ytwLDazWWejxLVn1lQ2FVjUQ7E-LQdr51AnAjR7CRQ_JV4BoLd1rHC7z_NLTsU1Fd8tNcAEgXuVOaTBandFhmCueR6M25G8hkHA/s1600/Screen+Shot+2020-05-24+at+7.59.43+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1125" data-original-width="1600" height="448" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMoQ6txi5I1vl7LSSbRalsmqVw8vQC-QqLV_6ytwLDazWWejxLVn1lQ2FVjUQ7E-LQdr51AnAjR7CRQ_JV4BoLd1rHC7z_NLTsU1Fd8tNcAEgXuVOaTBandFhmCueR6M25G8hkHA/s640/Screen+Shot+2020-05-24+at+7.59.43+PM.png" width="640" /></a></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Some example visualization using Workbooks on the Log Analytics Workspace to which we pump the data are as follows. We are still tweaking these and once we are in public preview I plan to come back and edit this post with links to public docs.<br />
<br />
In the screenshot below the visualization shows all database clusters of our test cluster at the same place. Selecting any cluster further drills down into each DB node health.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJwvQH8hktF-2ZMF732BfeFemyAciZCK0WQtngDKK1lIsI29gwg80WUel6cxjtiAkthr-Gxc9rw2PpwmnYdAa4nVrwzHqy1Pfp5J1_8lcIuZ_lcyaI-TIhxtvBbtPDNs4zKherOg/s1600/cluster.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1234" data-original-width="1266" height="622" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJwvQH8hktF-2ZMF732BfeFemyAciZCK0WQtngDKK1lIsI29gwg80WUel6cxjtiAkthr-Gxc9rw2PpwmnYdAa4nVrwzHqy1Pfp5J1_8lcIuZ_lcyaI-TIhxtvBbtPDNs4zKherOg/s640/cluster.PNG" width="640" /></a></div>
<br />
Similarly in the following visualization we see a cluster is unhealthy and then on drilling down a node is yellow (warning state) because it is triggering our high CPU usage threshold (>50%).<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix3INA4nYfJMlJaadn2By3TVbbeuJlrN8PzeFHS5lQzgjqJFbtilLCPHHdZT8j5370jWBj2B1Yw3oKvi3VBtoz7WEiI4hx_LiOwNXN2QL0FzJ_Gq-r7r67BEBuMOSN0hqg1bNbgw/s1600/cluster2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1379" data-original-width="1395" height="632" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix3INA4nYfJMlJaadn2By3TVbbeuJlrN8PzeFHS5lQzgjqJFbtilLCPHHdZT8j5370jWBj2B1Yw3oKvi3VBtoz7WEiI4hx_LiOwNXN2QL0FzJ_Gq-r7r67BEBuMOSN0hqg1bNbgw/s640/cluster2.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<h2 style="text-align: left;">
</h2>
<h2 style="text-align: left;">
Architecture </h2>
<div>
<div style="background-color: white; border-top: 1px solid rgb(238, 238, 238); color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px; padding-top: 8px;">
Our product is built on Kubernetes (or rather Azure Kubernetes Service), helm, linkerd, go-lang, fluentd and similar open source software. We use the engineering principles outlined <a href="https://blog.bonggeek.com/2020/02/system-engineering-guidelines.html">here</a>. Also we stand on the shoulder of giants, we did not have to build many core functionality because it comes for free inside the Azure engineering umbrella. We simply onboard to internal services that provide RBAC, cross region load balancing, billing etc.<br />
<br />
If the architecture seems familiar it is because a large part of it is shared with how we manage BareMetal blades running in memory databases (HANA) in Azure and I have posted about that <a href="https://blog.bonggeek.com/2020/04/managing-baremetal-blades-in-azure.html">here</a>.</div>
<div style="background-color: white; color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<br /></div>
<div style="background-color: white; color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
At the high level our architecture looks as follows.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL8mU_lUcyU6Ggf2EW7b62cOCVzrqKuNEgJp6z9s2J3jlCiTIMH98ES7MtKo36XrbloUf2S856yyOhBVfWCKbWrt8S8rdRRzmoSVFgB2bluJWKZR579zszYSesrWrZSJy6a0Xi5w/s1600/Screen+Shot+2020-05-24+at+7.53.18+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1238" data-original-width="1142" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL8mU_lUcyU6Ggf2EW7b62cOCVzrqKuNEgJp6z9s2J3jlCiTIMH98ES7MtKo36XrbloUf2S856yyOhBVfWCKbWrt8S8rdRRzmoSVFgB2bluJWKZR579zszYSesrWrZSJy6a0Xi5w/s640/Screen+Shot+2020-05-24+at+7.53.18+PM.png" width="590" /></a></div>
<br /></div>
</div>
<div style="background-color: white;">
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
The user/customer interacts with out system using either the Azure Portal (screenshot above), the command line tools or the SDK. We build extensions to the Azure portal for our product sub-area. All resources in azure is exposed using standardized RESTful APIs. The swagger spec is published <a href="https://github.com/Azure/azure-rest-api-specs/tree/master/specification/hanaonazure" style="color: #4d469c; text-decoration-line: none;" target="_blank">here</a> and the CLI and SDK is generated out of those.</div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<br /></div>
<div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
All interactions of the customer is handled first by the central Azure Resource Manager (ARM). It handles authentication and RBAC. Every resource type in Azure is handled by a corresponding resource provider. In this particular case the resource is Microsoft.HanaOnAzure/sapMonitors and it is handled by the HANA-RP (also referred to as just RP for simplification in this post). ARM knows to forward calls that it gets from customers to a particular regional instance of HANA-RP after taking care of authentication and other gate keeper activities.</div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<span style="font-size: 14px;"><br /></span></div>
<h3 style="text-align: left;">
<span style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif;"><span style="font-size: 14px;">The regional Resource Provider or RP</span></span></h3>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<span style="font-size: 14px;">For every Azure region we support we have a HANA-RP (resource provider or RP) instance deployed in that region. The RP is a collection of services that runs on </span><a href="https://docs.microsoft.com/en-us/azure/aks/">Azure Kubernetes Service (AKS)</a>. HANA-RP is build mostly using go-lang and engineered through Azure DevOps. We have automated build pipeline for the RP and single click (maybe a few clicks) deployment. We use use <a href="https://helm.sh/">Helm</a> for management.</div>
</div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<br /></div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
The service itself is stateless and the state is stored externally in Azure SQL Server. We use both structured data and document-DB style data. All data is replicated remotely to one more region, we configure automated backups for disaster recovery scenarios.</div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<br /></div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
We do not share any state across the RP instances. This provides an important attribute we look for in Azure services, regional isolation. This ensures that in case there is a regional Azure outage it does not effect any other regions.</div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<br /></div>
<div style="color: #444444; font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
Each instance of RP manages all monitors in its region. When the user uses the CLI/Portal to create the monitor all the details flow over encrypted channel from the ARM into the RP. The RP then deploys the monitoring payload into customers vnet.</div>
<div style="font-family: arial, tahoma, helvetica, freesans, sans-serif; font-size: 13px;">
<div style="color: #444444;">
<br /></div>
<span style="color: #444444;">All data flow across pods (intra service) and across the services are encrypted in transit and the data that we store is SQL Server is also encrypted at rest. We do not store any customer secrets on our systems (more below).</span><br />
<div style="color: #444444;">
<br /></div>
<div style="color: #444444;">
Tech usage: AKS, Kubernetes, linkerd, nginx, helm, linux, Docker, go-lang, Python, SQL Server, Azure DevOps, Azure Container Repository, Azure Key-vault, etc.</div>
</div>
</div>
<h3 style="text-align: left;">
Deployment</h3>
<div>
Once the RP gets a request to provision a monitor, it talks to other Azure resource providers like Compute, Storage, Security, Networking to setup the monitoring payload inside the customers vnet</div>
<div>
<ol style="text-align: left;">
<li>The RP creates various networking components (NSG, NIC)</li>
<li>Creates storage account, storage queues</li>
<li>Uses KeyVault to deploy DB access secrets. These are not stored by us, they remain encrypted in transit and in rest inside customer owned KeyVault</li>
<li>Creates log-analytics workspace</li>
<li>Creates collector VM in the resource group (a VM of type <a href="https://azure.microsoft.com/en-us/blog/introducing-b-series-our-new-burstable-vm-size/" target="_blank">B2ms</a>) </li>
<li>The VM uses custom script extension to bootstrap docker and pulls down the monitoring payload docker image</li>
</ol>
</div>
<h3 style="text-align: left;">
The Payload</h3>
<div>
Since the payload runs inside the customers vnet, we want to be absolutely transparent about what runs inside it. The entire payload is open source and can be accessed at <a href="https://github.com/Azure/AzureMonitorForSAPSolutions">https://github.com/Azure/AzureMonitorForSAPSolutions</a>. Specifically at <a href="https://github.com/Azure/AzureMonitorForSAPSolutions/tree/master/sapmon/payload">https://github.com/Azure/AzureMonitorForSAPSolutions/tree/master/sapmon/payload</a>.</div>
<br />
The commands use to install, launch and manage individual sub-monitors is in <a href="https://github.com/Azure/AzureMonitorForSAPSolutions/blob/master/sapmon/payload/sapmon.py" target="_blank">sapmon.py</a>. Specific payloads are in say <a href="https://github.com/Azure/AzureMonitorForSAPSolutions/blob/master/sapmon/payload/provider/saphana.py">saphana.py</a> or other files in that folder.<br />
<br />
Our payload VM fetches the docker image built out of these sources from our Azure container repository from the following location<br />
<span style="background-color: #fffffe; color: #0451a5; font-family: consolas, "courier new", monospace; font-size: 12px; white-space: pre;">mcr.microsoft.com/oss/azure/azure-monitor-for-sap-solutions</span><br />
<div style="text-align: left;">
Once this payload starts running inside the payload-VM, it fetches database connectivity information from customer's key-vault where the RP has placed that information. It then starts querying the database to fetch various monitoring information and pumping it into the Azure telemetry pipeline. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If the customer had opted-in during the monitor creation, the monitor also sends non identifiable telemetry back to Microsoft, so that we can ensure that the monitoring keeps functioning.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
We intentionally chose a design where the monitor does not run on the database machine itself and it is isolated in a separate VM. This ensures it is easy to observe the execution of the monitor and it is easy to isolate any impact it may have on the production system of the customer.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The way our monitoring is designed (execute monitoring queries against the database to fetch monitoring information) allows it to monitor any database that is reachable from inside the customers vnet. This includes obviously databases deployed on VMs inside the vnet. In addition it can monitor customer's HANA Large Instances that are running in BareMetal blades in VLANs that are accessible over express-route. Essentially as long as the database server name is resolvable and the database on it is reachable, the monitoring system works.</div>
<h3 style="text-align: left;">
Scalability</h3>
<div>
Our HANA-RP is automatically sharded by regions as it only handles all monitors in it's own region. Our stateless micro-services in each of those regions ensures we can easily horizontally scale to handle more control plane calls on the monitor in that region (create/delete monitors).</div>
<div>
<br /></div>
<div>
For the data-plane we actually deploy the entire payload in separate payload VMs inside the customer subscription/resource-group. So each new monitor comes with its own payload VM that monitors a DB (or a few instances of DB) for a given customer resulting in automatically scaling. The data also gets pumped into customer specific analytic workspaces and hence is not a bottleneck.</div>
<div>
<br /></div>
<div>
<ol style="text-align: left;">
</ol>
</div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-80022788938907500232020-05-05T11:58:00.001-07:002020-05-05T12:00:26.208-07:00Using Visual Studio Codespaces<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilbcbKA9QUT0L7xIQnSBtDdSZH520g-PDjtwLFJ4gFdNa2BXca58kZRNfKDdW7GDUF-72hz2l467KQGIHHAMUi1PyRr9aFiUps_xW4MyupONdvxM0sHvgYW8f7iOw31Ty9FhJMfQ/s1600/IMG_2647-Pano-Edit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1222" data-original-width="1600" height="488" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilbcbKA9QUT0L7xIQnSBtDdSZH520g-PDjtwLFJ4gFdNa2BXca58kZRNfKDdW7GDUF-72hz2l467KQGIHHAMUi1PyRr9aFiUps_xW4MyupONdvxM0sHvgYW8f7iOw31Ty9FhJMfQ/s640/IMG_2647-Pano-Edit.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<br />
One of the pain points we face with remote development is having to go through few extra hops to get to our virtual dev boxes. Many of us uses Azure VMs for development (in addition to local machines) and our security policy is to lock down all VMs to our Microsoft corporate network.<br />
<br />
So to ssh or rdp into an Azure VM for development, we first connect over VPN to corporate network, then use a corpnet machine to then login to the VMs. That is painful and more so now when we are working remotely.<br />
<br />
This is where the newly announced <a href="https://devblogs.microsoft.com/visualstudio/introducing-visual-studio-codespaces/" target="_blank">Visual Studio Codespaces</a> come in. Basically it is a hosted vscode in the cloud. It runs beautifully inside a browser and best of all comes with full access to the linux shell underneath. Since it is run as a service and secured by the Microsoft team building it, we can simply use it from a browser on any machine (obviously over two-factor authentication).<br />
<br />
At the time of writing this post, the cost is around $0.17 per hour for 4 core/8GB which brings the price to around $122 max for the whole month. Codespaces also has a snooze feature. I use snooze after one hour of no usage. This does mean additional startup time when you next login, but saves even more money. In between snooze the state of the box is retained.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQGm3A9P1RLDlbSG_8P0GPXPBQTG34wDSUnM-vubojVHr8PByat8UyMfTKi1WTgjTgZXIBNOD1OFVYr9v6nwcN0UHw6lTsS_z-iFPoRMNukqDmkuhE6a2g8QAi2sCmb3wahp2qng/s1600/Capture.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1114" data-original-width="1600" height="444" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQGm3A9P1RLDlbSG_8P0GPXPBQTG34wDSUnM-vubojVHr8PByat8UyMfTKi1WTgjTgZXIBNOD1OFVYr9v6nwcN0UHw6lTsS_z-iFPoRMNukqDmkuhE6a2g8QAi2sCmb3wahp2qng/s640/Capture.JPG" width="640" /></a></div>
<br />
While just being able to use the IDE on our code base is cool in itself, having access to the shell underneath is even cooler. Hit Ctrl+` in vscode to bring up the terminal window.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDj2aSROULRnwEreC1PJrPU09Q1vErEl5oM-lncLHIkfc0WCa5hzfjO791R_tLCO37vkWJatiA4f30i__BfR81hPqe8aXzu1noV1sp1s-so5v5dAcimrH-MbvFC11aBY1q9FUtVA/s1600/Capture2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="362" data-original-width="1214" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDj2aSROULRnwEreC1PJrPU09Q1vErEl5oM-lncLHIkfc0WCa5hzfjO791R_tLCO37vkWJatiA4f30i__BfR81hPqe8aXzu1noV1sp1s-so5v5dAcimrH-MbvFC11aBY1q9FUtVA/s640/Capture2.JPG" width="640" /></a></div>
<br />
I then sync'd my linux development environment from <a href="https://github.com/abhinababasu/share">https://github.com/abhinababasu/share</a>, installed required packages that I need. Finally I have a full shell and IDE in the cloud, just the way I want it.<br />
<br />
To try out Codespaces head to <a href="https://aka.ms/vso-login" rel="noreferrer noopener" style="font-family: "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", sans-serif; font-size: 14px;" tabindex="-1" target="_blank" title="https://aka.ms/vso-login">https://aka.ms/vso-login</a></div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-46746767476078442472020-04-26T02:48:00.002-07:002020-04-30T11:02:36.939-07:00Managing Baremetal blades in Azure<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTha6LBi2_J-crl9vbaO_LHUw3Lqas3H87WmNy2hGmwsD_faKHLGM1NoPpm1bLEwxs4uvzhrLHpfdPCAEPDmkV-uTdJuEcZV7pIjXt1fBvByWnYQ1qE0OWlDLClrDXiHhtL9_DnA/s1600/IMG_5421-Edit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1067" data-original-width="1600" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTha6LBi2_J-crl9vbaO_LHUw3Lqas3H87WmNy2hGmwsD_faKHLGM1NoPpm1bLEwxs4uvzhrLHpfdPCAEPDmkV-uTdJuEcZV7pIjXt1fBvByWnYQ1qE0OWlDLClrDXiHhtL9_DnA/s640/IMG_5421-Edit.jpg" width="640" /></a></div>
<br />
In this post I give a brief overview of how we run the control plane of BareMetal Compute in Azure that powers the SAP HANA Large instances (in memory database running on extreme high memory machines). We support different types of BareMetal blades that go all the way up to 24TB RAM including special memory support like Intel Optane (persistent memory) and unlike Virtual Machines, customer get full access to the BareMetal physical machine with root access but still behind network level security sandboxing.<br />
<br />
Few years back when we started the project we faced some daunting challenges. We were trying to get custom build SAP HANA certified bare-metal machines into Azure DCs, fit them in standard Azure racks and then manage them at scale and expose control knobs to customer inside Azure Portal. These are behemoths going up to 24 TB ram and based on size different OEMs were providing us the blades, storage, networking gear and fiber-channel devices.<br />
<br />
We quickly realized that most of the native Azure native compute stack will not work because they are built with design assumptions that do not hold for us.<br />
<ol style="text-align: left;">
<li>Azure fleet nodes or blades are built to Microsoft specification and have common denominator management API surface and monitoring, but we were bringing in disparate externally certified HW that did not meet us there</li>
<li>Our model needed to provide customer with full root access to the bare metal blades and they were not isolated across an hyper-visor</li>
<li>The allocator and other logic in Azure was not location aware. E.g. We had custom NetApp storage literally placed beside the high memory compute for very low latency, high throughput usage that is required by the SAP HANA in memory databases</li>
<li>We had storage and networking requirements in terms of uptime, latency and throughput that were not met by standard Azure storage, latency and hence we had to build our own.</li>
<li>We differed in basic layout from Azure, e.g. our blades do not have any local storage and everything runs off remote storage, we had different NW topology (many HW NICs per blade with very different I/O requirements).</li>
</ol>
<div>
Obviously we had to re-build the full cloud stack but with much more limited resources. Instead of 100s and 1000s of engineers we had a handful. So we set down a few guiding principles</div>
<div>
<ol style="text-align: left;">
<li>Be frugal on resourcing</li>
<li>Rely on external services instead of trying to build in-house</li>
<li>Design for maintainability</li>
</ol>
<div>
Finally 3 years in, we can see that many of our decisions and designs are holding through the test of time. We have expanded to 10s of regions around the world, added numerous scenarios but at the same time never had to significantly scale our dev resources. </div>
<h2 style="text-align: left;">
Architecture</h2>
<div>
While it might seem obvious to lot of people building these kinds of services, it was an unlikely choice for a Microsoft service. Our stack is built on Kubernetes (or rather Azure Kubernetes Service), go-land, fluentd and similar open source software. Also we stand on the shoulder of giants, we did not have to invent many core areas because it comes for free inside Azure, like RBAC, cross region balancing etc. </div>
<div>
<br /></div>
<div>
At the high level our architecture looks as follows</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgD_spOLf4Chyphenhyphen9WMRKkEgi5Lt-FZNfRhtVpzPp24mZjGl4Q621fH-S7F5FrEAUe0rafEGhr6u1mDHAMykyfzQg_jGJWDuBl_IG1kLmO1OaQxYRW24asr-u-2PQiS0xXWTujRpmHAw/s1600/Screen+Shot+2020-04-26+at+2.44.41+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1230" data-original-width="1520" height="517" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgD_spOLf4Chyphenhyphen9WMRKkEgi5Lt-FZNfRhtVpzPp24mZjGl4Q621fH-S7F5FrEAUe0rafEGhr6u1mDHAMykyfzQg_jGJWDuBl_IG1kLmO1OaQxYRW24asr-u-2PQiS0xXWTujRpmHAw/s640/Screen+Shot+2020-04-26+at+2.44.41+AM.png" width="640" /></a></div>
</div>
<h3 style="text-align: left;">
Customer Experience</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGF1oA5HJi9GernBgUTI4zy_lgXoGZpUZeGJ1gUp6pGY5dKIDCkLj_RaSqvA30nZXWjo6yLFEive5DoRccAeGf7H-MuWHKclGKu2jLrvXi8WoXcSX4MhyhMpCTYx3TQyklPxQikw/s1600/Screen+Shot+2020-04-26+at+2.31.57+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1003" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGF1oA5HJi9GernBgUTI4zy_lgXoGZpUZeGJ1gUp6pGY5dKIDCkLj_RaSqvA30nZXWjo6yLFEive5DoRccAeGf7H-MuWHKclGKu2jLrvXi8WoXcSX4MhyhMpCTYx3TQyklPxQikw/s640/Screen+Shot+2020-04-26+at+2.31.57+AM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br />
The customer interacts with out system using either the Azure Portal (screenshot above), the command line tools or the SDK. We build extensions to the Azure portal for our product sub-area. All resources in azure is exposed using standardized RESTful APIs. We publish the swagger spec <a href="https://github.com/Azure/azure-rest-api-specs/tree/master/specification/hanaonazure" target="_blank">here</a> and the CLI and SDK is generated out of those.</div>
<div>
<br /></div>
<div>
In any case all interactions of the customer is handled by the central Azure Resource Manager (ARM). It handles authentication, RBAC, etc. Every resource type in Azure is handled by a corresponding resource provider. In our case it is the HANA or BareMetal RP (BmRP). ARM knows (via data the BmRP provides back to it) how to forward calls that it gets from customers to a particular regional instance of BmRP.</div>
<h2 style="text-align: left;">
The regional Resource Provider or RP</h2>
<div>
If we are in N Azure regions then the BmRP (resource provider) is deployed in N instances (one in each region) and it runs on Azure Kubernetes Service (AKS). BmRP is build mostly using go-lang and engineered through Azure DevOps. We have automated build pipeline for the RP and single click (maybe a few clicks) deployment. We use use Helm to manage our deployment.</div>
<div>
<br /></div>
<div>
The service itself is stateless and the state is stored in SQL Server Azure. We use both structured data and document-DB style json. All data is replicated remotely to one more region, we configure automated backups for disaster recovery scenarios.</div>
<div>
<br /></div>
<div>
We do not share any state across the RP instances. We are particular about ensuring that every regional instance can completely work on its own. This is to ensure that in case there is a regional outage it does not effect any other regions.</div>
<div>
<br /></div>
<div>
Each instance of RP in turn manages multiple clusters of bare-metal machines. There is one or more such clusters per RP instances. Each cluster is managed by an instance of a cluster manager (CM). All communication between RP and the cluster manager is via two Azure service-bus-queue (SBQ). One from the BmRP to the CM and the other in reverse direction. BmRP issues various commands (JSON messages) to the CM through the SBQ and gets back responses from the CM via the other SBQ.</div>
<div>
<br />
We pump both metrics (hot-path) and logs (warm path) into our Azure wide internal telemetry pipeline called Jarvis. We then add backend alerts and dashboards on the metrics for near realtime alerting and in some cases also on the logs (using logs-to-metrics). The data is also digested into <a href="https://docs.microsoft.com/en-us/connectors/kusto/" target="_blank">Azure Kusto</a> (aka Data Explorer) which is a log analytics platform. The alerts tells us if something has gone wrong (severity two and above alerts ring on-call phones) and then we use the logs in Jarvis or Kusto queries to debug.<br />
<br />
Also it's good to call out that all control flow across pods and across the services is encrypted in transit over nginx and linkerd. The data that we store is SQL Server is also obviously encrypted in transit and at rest.<br />
<br />
Tech usage: AKS, Kubernetes, linkerd, nginx, helm, linux, Docker, go-lang, Python, Azure Data-explorer, Azure service-bus-queue, Azure SQL Server, Azure DevOps, Azure Container Repository, Azure Key-vault, etc.</div>
<h2 style="text-align: left;">
Cluster Manager</h2>
We have a cluster-manager (CM) per compute cluster. The cluster manager runs on AKS. The AKS vnet is connected over Azure Express Route into the management VLAN for the cluster that contains all our compute, storage and networking devices. All of these devices are in a physical cluster inside Azure Data-center.<br />
<br />
We wanted to ensure that the design is such that the cluster manager can be implemented with a lot of versatility and can evolve without dependency on the BmRP. The reason is we imagined one day the cluster-manager could also run inside remote locations (edge sites) and we weren't sure if we can call into the CM from outside or have some other sort of persistent connectivity. So we chose Azure Service Bus Queue for communication using simple json command response going between BmRP and CM. This only requires that the CM can make GET calls on the SBQ end point and nothing more to talk to BmRP.<br />
<br />
The cluster manager has two major functions<br />
<br />
<ol style="text-align: left;">
<li>Provide a device agnostic control abstraction layer to the RP</li>
<li>Monitor various devices (compute, storage and NW) in the cluster</li>
</ol>
<h3 style="text-align: left;">
Abstraction</h3>
<div>
Instead of having our BmRP know specifics of all types of HW in the system, it works on an abstraction. It expects basic generic CRUD type of operations being available on those devices and issues generic commands which then the cluster-manager translates to device specific actions.</div>
<div>
<br /></div>
<div>
It is easier to follow through how things work if we take one specific user workflow. Say a customer wants to reboot their BareMetal blade for some reason (an Update category operation). For this the customer hits the reboot button for their blade in Azure Portal, the Portal calls into Azure Resource Manager (ARM). ARM calls into BmRP's REST Api for the same. BmRp drops the reboot blade command into the service bus queue. Finally one of the replica of the cluster-manager container that is listening for those commands pick the command up. Now for various memory sizes and types we use different blades supplied by different OEMs. Some of those blades can be controlled remotely via Redfish APIs, some support ipmi commands, some even proprietary REST apis. The job of the cluster-manager is to take the generic reboot command, identify the exact type of the blade for which the command is and then issue blade specific control commands. It then takes the response and sends it back to the RP as an ack.</div>
<div>
<br /></div>
<div>
Similarly for storage it can handle a get-storage-status command by servicing it with ONTAP (NetApp storage manager) REST Api call .</div>
<h3 style="text-align: left;">
Telemetry and Monitoring</h3>
We use a <a href="https://www.fluentd.org/" target="_blank">fluentd</a> based pipeline for actively monitoring all devices in the cluster. We use active monitoring, which means that not only we listen onto the various events generated by the devices in the cluster, we also call into these devices to get more information.<br />
<br />
The devices in our cluster uses various types of event mechanism. We configure these various types of events like syslog, snmp, redfish events to be sent to the load-balancer of the AKS cluster. When one of the fluentd end-points get the event it is send through a series of input plugins. Some of the plugins filter out noisy events we do not care about, some of the plugins call back into the device that generated the event to get more information and augment the event.<br />
<br />
Finally the output plugins send the result into our Jarvis telemetry pipeline and other destinations. Many of the plugins we use or have built are open sourced, like <a href="https://github.com/Azure/fluent-plugin-process-ucs-syslog" target="_blank">here</a>, <a href="https://github.com/Azure/fluent-plugin-process-snmptrap" target="_blank">here</a>, <a href="https://github.com/Azure/fluent-plugin-hanarp-message" target="_blank">here</a> and <a href="https://github.com/Azure/fluent-plugin-servicebus-queue/" target="_blank">here</a>. Some of the critical event like blade power-state change (reboot) is sent back also through the service-bus-queue to the BmRP so that it can store blade power-state information in our database.<br />
<br />
Generally we rely on events being sent from devices to the fluentd end point for monitoring. Since many of these events come over UDP and the telemetry pipeline itself is running on a remote (from the actual devices) AKS cluster we expect some of the events to get dropped. To ensure we have high reliability on these events, we also hence have backup polling. The cluster manager in periodic intervals reaches out to the equipment in the cluster using whatever APIs those equipment support to get their status and fault-state (if any). Between the events and backup polling we have both near real-time as well as reliable coverage.<br />
<br />
Whether from events or from polling all telemetry is sent to our Jarvis pipeline. We then have Jarvis alerts configured on these events. E.g. if a thermal event occurs and either storage nodes or blade's temperature goes over a threshold the alert will fire. Same thing for cases like a blade crashing due to HW issues.<br />
<br />
Tech-user: AKS, Kubernetes, linux, Docker, fluentd, Ruby, go-lang, Python, Azure service-bus-queue, snmp, ipmi, Redfish, ONTAP, syslog, etc.<br />
<h2 style="text-align: left;">
Scaling and Reliability</h2>
<div>
Our front-door is Azure Resource Manager (ARM) through which all users come in. ARM has served us well and provides us with user-authentication, throttling, caching and regional load balancing. It forwards user calls for blades in one region to the RP in that region. So as we add more regions we simply deploy a new instance of the BmRP for that region, register with ARM and scale. </div>
<div>
<br /></div>
<div>
Even in the same region as we land more infra we put them in new clusters with about 100-ish blades and its corresponding storage. Since each cluster has its own cluster manager with a shared nothing model, the cluster manager scales as well along with every cluster we land. As we add more scenarios in the cluster manager though we need to scale the manager itself.</div>
<div>
<br /></div>
<div>
Scaling the cluster manager itself is also trivial in our design, it either gets event traffic from devices or commands from BmRP over SBQ that it then executes. The events come through at the load-balancer of the AKS cluster and hence just increasing the replica count of those containers work. Commands coming from the BmRP arrives over service bus queue and all these containers listen on the same queue, so when we add more replicas of these containers the worker count goes up.</div>
<div>
<br /></div>
<div>
Since we use separate SBQ per CM, adding new CM automatically means new SBQ pair gets created for it. The only concern could be that we add so many new scenarios in the CM that the traffic between one CM and BmRP goes up enough to cause bottleneck in the SBQ. However, SBQ itself can be scaled up to handle it (we have never had to do that), or worst case we may have to add more SBQ, sharding by types of commands and scale it horizontally. To be honest this is not something I think will ever happen.</div>
<div>
<br /></div>
<div>
With all of the above we have been able to meet 99.95 uptime for our control plane and keep latencies under our target. The only place where we hit issues is connectivity with our SQL DB. We had to upscale the DB SKU in the past. We continue to infrequently hit timeouts and other issues at the DB. At one point we were talking about moving to CosmosDB as it is touted to be more reliable, but most likely we will invest in some sort of caching in the future. That can either be by deploying some sort of cache engine in the BmRP itself or most likely use Azure Redis Cache (see principles at the top of this post).</div>
<div>
</div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-57542315394832172612020-02-17T17:30:00.000-08:002020-02-25T07:56:58.122-08:00System Engineering Guidelines<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg78IBeUA9yI3h62nGxxoRR0jWtuxjxQj4_s3ACsGoINzebG6-B0nH5TdMZ9Mgb-Xjybck-8zFQIzRPmuge3Su9fouO02crQXxIry01X8b_AVytg_udOKhEgsfuw9QjzBD1tvauxQ/s1600/IMG_3830.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="755" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg78IBeUA9yI3h62nGxxoRR0jWtuxjxQj4_s3ACsGoINzebG6-B0nH5TdMZ9Mgb-Xjybck-8zFQIzRPmuge3Su9fouO02crQXxIry01X8b_AVytg_udOKhEgsfuw9QjzBD1tvauxQ/s640/IMG_3830.jpg" width="482" /></a></div>
<div>
<br /></div>
<div>
While building our system that powers memory intensive compute in Azure we use the following engineering guidelines. We use these guidelines to build our BareMetal resource provider, cluster manager etc. These are useful principles we have accumulated from experience building various systems. <i>What other principles do you use and recommend including?</i></div>
<ol style="font-size: 14px; text-align: left;">
<li class="code-line" data-line="10" style="position: relative;"><div class="code-line" data-line="10" style="position: relative;">
<strong>Close on broad design before sending PRs</strong>.</div>
<ol>
<li class="code-line" data-line="11" style="position: relative;">Add design as markdown to root of the feature code path and discuss it as a PR. For broad cross RP feature it is ok to place it in the root</li>
<li class="code-line" data-line="13" style="position: relative;">For sizable features please have a design meeting</li>
<li class="code-line" data-line="14" style="position: relative;">Requirement for a design meeting is to send a pre-read and expectation is all attendees have reviewed the pre-read before coming in<br />
</li>
</ol>
</li>
<li class="code-line" data-line="17" style="position: relative;"><div class="code-line" data-line="17" style="position: relative;">
<strong>Be aware of distributed system quirks</strong></div>
<ol>
<li class="code-line" data-line="18" style="position: relative;">Think CAP theorem. This is a distributed system, network partition will occur, be explicit about your availability and consistency model in that event</li>
<li class="code-line" data-line="20" style="position: relative;">All remote calls will fail, have re-tries that uses exponential back-off. Log warning on re-tries and error if it finally fails</li>
<li class="code-line" data-line="22" style="position: relative;">Ensure we always have consistent state. There should be only 1 authoritative version of truth. Having local data that is eventually consistent with this truth is acceptable. Know the max time-period for eventual consistency<br />
</li>
</ol>
</li>
<li class="code-line" data-line="26" style="position: relative;"><div class="code-line" data-line="26" style="position: relative;">
<strong>System needs to be reliable, scalable and fault tolerant</strong></div>
<ol>
<li class="code-line" data-line="27" style="position: relative;">Always avoid SPOF (Single Point of Failure), even for absolutely required resources like SQLServer consider retrying (see below), gracefully fail and recover</li>
<li class="code-line" data-line="29" style="position: relative;">Have retries</li>
<li class="code-line" data-line="30" style="position: relative;">APIs need to be responsive and return in sub second for most scenarios. If something needs to take longer, immediately return with a mechanism to track progress on the background job started</li>
<li class="code-line" data-line="32" style="position: relative;">All API and actions we support should have a 99.9 uptime/success SLA. Shoot for 99.95</li>
<li class="code-line" data-line="33" style="position: relative;">Our system should be stateless (have state elsewhere in data-store) and designed to be cattle and not pets</li>
<li class="code-line" data-line="34" style="position: relative;">Systems should be horizontally scalable. We should be able to simply add more nodes to a cluster to handle more traffic</li>
<li class="code-line" data-line="36" style="position: relative;">Choose to use a managed service over attempting to build it or deploy it in-house<br />
</li>
</ol>
</li>
<li class="code-line" data-line="38" style="position: relative;"><div class="code-line" data-line="38" style="position: relative;">
<strong>Treat configuration as code</strong></div>
<ol>
<li class="code-line" data-line="39" style="position: relative;">Breaks due to out of band config changes are too common. So consider config deployment the same way as code deployment (Use SCD == Safe Config/Code Deployment)</li>
<li class="code-line" data-line="41" style="position: relative;">Config should be centralized. Engineers shouldn't be hunting around to look for configs<br />
</li>
</ol>
</li>
<li class="code-line" data-line="43" style="position: relative;"><div class="code-line" data-line="43" style="position: relative;">
<strong>All features must have feature flag in config</strong>.</div>
<ol>
<li class="code-line" data-line="44" style="position: relative;">The feature flag can be used to disable features in per region basis</li>
<li class="code-line" data-line="46" style="position: relative;">Once a feature flag is disabled the feature should cause no impact to the system<br />
</li>
</ol>
</li>
<li class="code-line" data-line="48" style="position: relative;"><div class="code-line" data-line="48" style="position: relative;">
<strong>Try to make sure your system works on a single box</strong>This makes dev-test significantly easier. Mocking auxiliary systems is OK</div>
</li>
<li class="code-line" data-line="51" style="position: relative;"><div class="code-line" data-line="51" style="position: relative;">
<strong>Never delete things immediately</strong></div>
<ol>
<li class="code-line" data-line="52" style="position: relative;">Don't delete anything instantaneously, especially data. Tombstone deleted data away from user view</li>
<li class="code-line" data-line="53" style="position: relative;">Keep data, metadata, machines around for a garbage collector to periodically delete at configurable duration.<br />
</li>
</ol>
</li>
<li class="code-line" data-line="56" style="position: relative;"><div class="code-line" data-line="56" style="position: relative;">
<strong>Strive to be event driven</strong></div>
<ol>
<li class="code-line" data-line="57" style="position: relative;">Polling is bad as the primary mechanism</li>
<li class="code-line" data-line="58" style="position: relative;">Start with event driven approach and have fallback polling<br />
</li>
</ol>
</li>
<li class="code-line" data-line="60" style="position: relative;"><div class="code-line" data-line="60" style="position: relative;">
<strong>Have good unit tests</strong>.</div>
<ol>
<li class="code-line" data-line="61" style="position: relative;">All functionality needs to ship with tests in the same PR (no test PR later)</li>
<li class="code-line" data-line="62" style="position: relative;">Unit test tests functionality of units (e.g. class/modules)</li>
<li class="code-line" data-line="63" style="position: relative;">They do not have to test every internal functions. Do not write tests for tests' sake. If test covers all scenarios exposed by an unit, it is OK to push back on comments like "test all methods".</li>
<li class="code-line" data-line="65" style="position: relative;">Think what does your unit implement and can the test validate the unit is working after any changes to it</li>
<li class="code-line" data-line="66" style="position: relative;">Similarly if you add a reference to an unit from outside and depend on a behavior consider adding a test to the callee so that changes to that unit doesn’t break your requirements</li>
<li class="code-line" data-line="68" style="position: relative;">Unit test should never call out from dev box, they should be <em>local</em> tests only</li>
<li class="code-line" data-line="69" style="position: relative;">Unit test should not require other things to be spun up (e.g. local SQL server)<br />
</li>
</ol>
</li>
<li class="code-line" data-line="71" style="position: relative;"><div class="code-line" data-line="71" style="position: relative;">
<strong>Consider adding BVT to scenarios that cannot be tested in unit tests</strong>. <br />
E.g. stored procs need to run against real SqlDB deployed in a container during BVT, or test query routing that needs to run inside a web-server</div>
</li>
<li class="code-line" data-line="71" style="position: relative;"><b>All required tests should be automatically run</b> and not require humans to remember to run them</li>
<li class="code-line" data-line="75" style="position: relative;"><div class="code-line" data-line="75" style="position: relative;">
<strong>Test in production via our INT/canary cluster</strong>Somethings simply cannot be tested on dev setup as they rely on real services to be up. For these consider testing in production over our INT infra.</div>
<ol>
<li class="code-line" data-line="78" style="position: relative;">All merges are automatically deployed to our INT cluster</li>
<li class="code-line" data-line="78" style="position: relative;"><div class="code-line" data-line="78" style="position: relative;">
Add runners to INT that simulate customer workloads.</div>
</li>
<li class="code-line" data-line="78" style="position: relative;">Add real lab devices or fake devices that test as much as possible. E.g. add fake snmp trap generator to test fluentd pipeline, have real blades that can be rebooted using our APIs periodically</li>
<li class="code-line" data-line="78" style="position: relative;">Bits are then deployed to Canary clusters where there are real devices being used for internal testing, certification. Bake bits in Canary!</li>
</ol>
</li>
<li class="code-line" data-line="86" style="position: relative;"><div class="code-line" data-line="86" style="position: relative;">
<strong>All features should have measurable KPIs and metrics</strong>.</div>
<ol>
<li class="code-line" data-line="87" style="position: relative;">You must add metrics against new features. Metrics should tell how well your feature is working, if your feature stops working or if any anomaly is observed</li>
<li class="code-line" data-line="89" style="position: relative;">Do not skimp on metrics, we can filter metrics on the backend rather than not having them fired<br />
</li>
</ol>
</li>
<li class="code-line" data-line="91" style="position: relative;"><div class="code-line" data-line="91" style="position: relative;">
<strong>Copious logging is required</strong>.</div>
<ol>
<li class="code-line" data-line="92" style="position: relative;">Process should never fail silently</li>
<li class="code-line" data-line="93" style="position: relative;">You must add logs for both success and failure paths. Err on the side of too much logging<br />
</li>
</ol>
</li>
<li class="code-line" data-line="95" style="position: relative;"><div class="code-line" data-line="95" style="position: relative;">
<strong>Do not rely on text logs to catch production issues</strong>.</div>
<ol>
<li class="code-line" data-line="96" style="position: relative;">You cannot rely on too many error logs from a container to catch issues. Have metrics instead (see above)</li>
<li class="code-line" data-line="97" style="position: relative;">Logs are a way to root-cause and debug and not catch issues<br />
</li>
</ol>
</li>
<li class="code-line" data-line="99" style="position: relative;"><div class="code-line" data-line="99" style="position: relative;">
<strong>Consider on-call for all development</strong></div>
<ol>
<li class="code-line" data-line="100" style="position: relative;">Ensure you have metrics and logs</li>
<li class="code-line" data-line="101" style="position: relative;">Ensure you write good documentation that anyone in the team can understand without tons of context</li>
<li class="code-line" data-line="102" style="position: relative;">Add alerts with direct link to TSGs</li>
<li class="code-line" data-line="103" style="position: relative;">Add actionable alerts where the on-call can quickly mitigate</li>
<li class="code-line" data-line="104" style="position: relative;">On-call should be able to turn off specific features in case it is causing problems in production</li>
<li class="code-line" data-line="105" style="position: relative;">All individual merges can be rolled back. Since you cannot control when code snap for production happens the PRs should be such that it can be individually rolled back<br />
</li>
</ol>
</li>
</ol>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-10584558352633474302020-02-14T15:19:00.001-08:002020-02-25T07:57:19.762-08:00The C Word - Part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS_Wok7ywUWhQkMx6CF2gyY7lsT9dJKleAZzGYKXvSnO1MZ4asOIRkg23Qs3cwGvVCLDlUehljy0VeAVwTy3_TXYSZKEO0jpnmGGWb5My7_cCUBmuR3fbYrpDbZiaUmiU-1Pcb5A/s1600/IMG_4536-Edit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="667" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS_Wok7ywUWhQkMx6CF2gyY7lsT9dJKleAZzGYKXvSnO1MZ4asOIRkg23Qs3cwGvVCLDlUehljy0VeAVwTy3_TXYSZKEO0jpnmGGWb5My7_cCUBmuR3fbYrpDbZiaUmiU-1Pcb5A/s640/IMG_4536-Edit.jpg" width="426" /></a></div>
<br />
<br />
Our confrontation with Lymphoma started in 2011, when <a href="https://blog.bonggeek.com/2011/08/c-word.html" target="_blank">the C word entered</a> our life and my wife got diagnosed with Stage 4 Hodgkins sclerosing lymphoma. We have fought through and continue to do so. Even through she is in remission now, the shadow of Cancer still hangs over.<br />
<br />
We had just moved across the world from India to the US in 2010 and had no family and very few friends around. We had to mostly duke it out ourselves with little support. We were able to get the best treatment available in the world through the <a href="https://www.fredhutch.org/" target="_blank">Fred Hutch</a> and <a href="https://www.seattlecca.org/" target="_blank">Seattle Cancer Care Alliance</a>. However, we do realize not everyone is fortunate to be able to do so.<br />
<br />
Our daughter has decided to do her part now and raise funds through the <a href="https://www.lls.org/" target="_blank">Lymphoma and Leukemia Society</a>. If you'd like to help her please head to<br />
<h3 style="text-align: center;">
<a href="https://events.lls.org/wa/SOYSeattle20/pbasu">https://events.lls.org/wa/SOYSeattle20/pbasu</a></h3>
<br />
<br /></div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-37835892656749746602020-01-19T04:00:00.000-08:002020-01-19T17:34:28.462-08:00Chobi - Face Detection Based Static Image Gallery Generator<div dir="ltr" style="text-align: left;" trbidi="on">
Over the holidays I created a simple static photo gallery generator that I named chobi. The sources are in <a href="https://github.com/abhinababasu/chobi">https://github.com/abhinababasu/chobi</a>.<br />
<div>
<br /></div>
<div>
<div>
Given a source folder of photos, chobi will generate a destination folder containing the original photos, generated thumbnails, css stylesheets, scripts and html files which constitute a website displaying those photos as follows</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXMksZk2gRNEEMDqu-QfmBbbeBoqidp1WFdY1FYhL0QIrGYM3zIp1cg8pPFeR-j0zUdySc8AKlwex2O_2m6NCtfYAjRTQtG9GpdU-XTcPk-ZHI3uO-6g0ae42h2__0vWQCa8i2Q/s1600/sample2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="434" data-original-width="600" height="462" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXMksZk2gRNEEMDqu-QfmBbbeBoqidp1WFdY1FYhL0QIrGYM3zIp1cg8pPFeR-j0zUdySc8AKlwex2O_2m6NCtfYAjRTQtG9GpdU-XTcPk-ZHI3uO-6g0ae42h2__0vWQCa8i2Q/s640/sample2.jpg" width="640" /></a></div>
<div>
<br /></div>
<div>
<h2>
The Problem</h2>
<div>
While building chobi and also when I was looking into similar online tools, I kept hitting a major issue and that prompted further work and this post.<br />
<br />
You see thumbnail generation from image has a major problem. I needed the gallery generator to create square thumbnails for the image strip shown at the bottom of the page. However, the generated thumbnails would simply be either from the center or some other arbitrary location. This meant that the thumbnails would cut off at weird places.</div>
<div>
<br /></div>
<div>
<div>
Consider the following image.</div>
</div>
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOhLyM22QqT4o0Cu-lDeZ4x20OOZPp6anK3kawskoZAUnbfEx43u8yRDjVgN9wPkUlhYMzXIN1EZfD9UgRC78Zl76I07T_md5hgR9Jn1o4zTDkDpLsTUdyn_oxJvq3Wf8gW5TSFg/s1600/portrait1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="375" data-original-width="251" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOhLyM22QqT4o0Cu-lDeZ4x20OOZPp6anK3kawskoZAUnbfEx43u8yRDjVgN9wPkUlhYMzXIN1EZfD9UgRC78Zl76I07T_md5hgR9Jn1o4zTDkDpLsTUdyn_oxJvq3Wf8gW5TSFg/s200/portrait1.jpg" width="133" /></a></div>
<div>
<br /></div>
<div>
If I create a thumbnail from a tool without knowing where the face is in the image, it will generate something like</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD5MwLUvhmq1F9Ypudjjx1huEl7WJ8Hpo7cpp-YRRvMr0l5NqaH9j6TwYgnp0p__gyRMrdymGnhgMhZyjJ8OS3FQlghzVqpAf_OD0EKz3PW8yPFMDbSz9UfiacTzyy8xrKXlslfw/s1600/thumb1_1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="150" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD5MwLUvhmq1F9Ypudjjx1huEl7WJ8Hpo7cpp-YRRvMr0l5NqaH9j6TwYgnp0p__gyRMrdymGnhgMhZyjJ8OS3FQlghzVqpAf_OD0EKz3PW8yPFMDbSz9UfiacTzyy8xrKXlslfw/s1600/thumb1_1.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Obviously that doesn't work. So using my vanilla generator I got a website with all sorts of similar head chopped off thumbnails (marked in Red)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifFthHZGMt20xtySkfkhyE4ZsHtxDAlhXppTd_quxBIxNlzks1Ie8TCckEahRxbh2UhZZtvZ7Y67ntdhu0tqeRc03OW_MyyGOlpX64K1g5z1PaKl4-SBvjv5ymjMcUkLQxxKT87g/s1600/sample1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="435" data-original-width="600" height="464" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifFthHZGMt20xtySkfkhyE4ZsHtxDAlhXppTd_quxBIxNlzks1Ie8TCckEahRxbh2UhZZtvZ7Y67ntdhu0tqeRc03OW_MyyGOlpX64K1g5z1PaKl4-SBvjv5ymjMcUkLQxxKT87g/s640/sample1.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h2 style="text-align: left;">
The Solution</h2>
<div>
<div>
Chobi uses face-detection to ensure that does not happen and the face is always fully present in the generated thumbnail. Consider the same thumbnail as above but now with face-detection</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7iMu3u60knSoclzmLXDSzB5r45fbDNp-4ZZjuxjZ9Tf-P3_1lA55DL_KFVB4VjrbY0IoyRCQQF7Grf6ZN4Qpdnfk0yOrr1DOhUb_is1QO-w23BBcdpcRqqakeuNOfS6uk3CTzQ/s1600/thumb1_2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="150" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7iMu3u60knSoclzmLXDSzB5r45fbDNp-4ZZjuxjZ9Tf-P3_1lA55DL_KFVB4VjrbY0IoyRCQQF7Grf6ZN4Qpdnfk0yOrr1DOhUb_is1QO-w23BBcdpcRqqakeuNOfS6uk3CTzQ/s1600/thumb1_2.jpg" /></a></div>
<div>
<br /></div>
<div>
Another example of an original image and then generated thumbnail first without and then with face-detection.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPqrHkdJ0q2_zbychePfsbW8J1vKHiyGsWFyjOr0CtMV6GxAS7xiPAW_4ybsArnkWIEQI5sCU-kJcEQZaXprk_BUAXUG1UKItIwY0Kap9JF4vaK9FkvKWV8UijaM1_iWRo4-6C9g/s1600/portrait2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="438" data-original-width="250" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPqrHkdJ0q2_zbychePfsbW8J1vKHiyGsWFyjOr0CtMV6GxAS7xiPAW_4ybsArnkWIEQI5sCU-kJcEQZaXprk_BUAXUG1UKItIwY0Kap9JF4vaK9FkvKWV8UijaM1_iWRo4-6C9g/s320/portrait2.jpg" width="182" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDwRjWSmjBNU0flNh_XJZeyP80vaocD7v5G0EmeiIuk9uyTiXPOmAhRuMyY67QgCOCOyt0p1Pam3rdN4v94sEK1heH8bEhiDJoVBiLyfMOgx4asZqvqP7_nn7npqcghlbBwfzq0A/s1600/thumb2_1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="150" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDwRjWSmjBNU0flNh_XJZeyP80vaocD7v5G0EmeiIuk9uyTiXPOmAhRuMyY67QgCOCOyt0p1Pam3rdN4v94sEK1heH8bEhiDJoVBiLyfMOgx4asZqvqP7_nn7npqcghlbBwfzq0A/s1600/thumb2_1.jpg" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZwa8gx7y08P_oZhkoLIET7fiFeD62cIz9dirESMDNQU_TEmULDCc76IRm30xLWGS6mBLWO7MXUR0dgQaR-F4iEyU5zOWUb2Q-8SJ4G02nVB216l-jVN-tNZ8YZ78lSLXGMY1L9w/s1600/thumb2_2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="150" data-original-width="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZwa8gx7y08P_oZhkoLIET7fiFeD62cIz9dirESMDNQU_TEmULDCc76IRm30xLWGS6mBLWO7MXUR0dgQaR-F4iEyU5zOWUb2Q-8SJ4G02nVB216l-jVN-tNZ8YZ78lSLXGMY1L9w/s1600/thumb2_2.jpg" /></a></div>
<div>
<br /></div>
<div>
With this face-detection plugged in chobi generates a much better web-site, with almost no photo cropped where it shouldn't be.</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXMksZk2gRNEEMDqu-QfmBbbeBoqidp1WFdY1FYhL0QIrGYM3zIp1cg8pPFeR-j0zUdySc8AKlwex2O_2m6NCtfYAjRTQtG9GpdU-XTcPk-ZHI3uO-6g0ae42h2__0vWQCa8i2Q/s1600/sample2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="434" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXMksZk2gRNEEMDqu-QfmBbbeBoqidp1WFdY1FYhL0QIrGYM3zIp1cg8pPFeR-j0zUdySc8AKlwex2O_2m6NCtfYAjRTQtG9GpdU-XTcPk-ZHI3uO-6g0ae42h2__0vWQCa8i2Q/s1600/sample2.jpg" /></a></div>
<div>
<br /></div>
<h2 style="text-align: left;">
Sources</h2>
<div>
<ol style="text-align: left;">
<li>Chobi sources are at <a href="https://github.com/abhinababasu/chobi">https://github.com/abhinababasu/chobi</a></li>
<li>It uses the face-detected thumbnail generator which I wrote at <a href="https://github.com/abhinababasu/facethumbnail">https://github.com/abhinababasu/facethumbnail</a>. </li>
<li>That is in turn is based out of a face detection library <a href="https://github.com/esimov/pigo">pigo </a>written fully in go</li>
</ol>
<h2 style="text-align: left;">
Sample</h2>
</div>
</div>
<div>
Checkout a gallery build using chobi at </div>
<div>
<a href="http://bonggeek.com/Photography/People.html">http://bonggeek.com/Photography/People.html</a></div>
<div>
<br /></div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-70335662155692197122020-01-12T17:30:00.000-08:002020-01-12T17:30:00.140-08:00How to run Windows 7 after end of support<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiJWh2HKxnxTINsXB32tvJ7t9osV7reMQWBhEtRDwT5f6u06wE33GgqzwoQqwj_1mmnjuvq_rOGnA0W8TaoXJXPWY1BSJ6KzKS5f-aakXMtjyf7ygSxHtxBD_0GTghf1a-3vzzmw/s1600/53.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="911" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiJWh2HKxnxTINsXB32tvJ7t9osV7reMQWBhEtRDwT5f6u06wE33GgqzwoQqwj_1mmnjuvq_rOGnA0W8TaoXJXPWY1BSJ6KzKS5f-aakXMtjyf7ygSxHtxBD_0GTghf1a-3vzzmw/s640/53.jpg" width="582" /></a></div>
<a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br />
<br />
<br />
Windows 7 end of support is upon us in 1 more day (1/14/2020). This post tries to answer the question on whether you can safely continue to run it. The short answer is that you can't, atleast if it is connected to the outside in some form.<br />
<br />
However, I have a friend back in India who has some software that he relies on and he can't run it on modern Windows. So when I was answering his question on how he can run it, I thought I'd write it up in the blog as well.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>
This post outlines how you can run Windows 7 in virtual machine running on Microsoft Hyper-visor on a windows 10 machine. The process also uses checkpoints to reset the VM back to the old state each time. This ensures that even if something malicious gets hold of the system, you can simple go back to the pristine state you started with.<br />
<h2 style="text-align: left;">
Pre-requisite</h2>
<div>
Obviously you need a computer capable of running Hyper-V. For my purpose I am using a Windows 10 Professional machine. You should also have more than enough CPU cores and memory to run Windows 7 in that machine. I recommend atleast 4 cores and 8GB memory so that you can give half of that to the Windows 7 VM and keep the rest for a functional host PC.<br />
<br />
Get hold of Windows 7 ISO or download it from <a href="https://www.microsoft.com/en-us/software-download/windows7">https://www.microsoft.com/en-us/software-download/windows7</a>.</div>
<div>
<br /></div>
<div>
Then visit the system requirement <a href="https://support.microsoft.com/en-us/help/10737/windows-7-system-requirements">https://support.microsoft.com/en-us/help/10737/windows-7-system-requirements</a>. I decided to give roughly twice the resources as the requirements to create my VM.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
Setup VM</h2>
<div>
Hit windows-key and type hyper-v to launch. Then start creating a VM by clicking New and then Virtual machine.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBXqgRHwM9yD0ENyF_Gmjdc1NGi9HJMJNgMw-_FeuMav3Th4Hi1i5Wje84mwpfWFk0MTb4-epzbV8MSmjdirtinQJfWIll4FjCeUEaSdtYmN8lVRHSfFRYbp2bgoFqYq4dXzIYpQ/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="176" data-original-width="680" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBXqgRHwM9yD0ENyF_Gmjdc1NGi9HJMJNgMw-_FeuMav3Th4Hi1i5Wje84mwpfWFk0MTb4-epzbV8MSmjdirtinQJfWIll4FjCeUEaSdtYmN8lVRHSfFRYbp2bgoFqYq4dXzIYpQ/s640/1.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br />
Choose the following Generation 1<br />
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLsTmTk68bHDzqIsGn3PLyyJk1KZVR-qswx-sHYwT5qorHdH2yDozhyJPeZ9aGtodyHCA_HwmrRur3OHhnKs_xHgWlwJLsJcTxFS3oQkoFOKFMcVKrIPCnnk3giLzCE9MjvJsd8A/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="704" height="484" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLsTmTk68bHDzqIsGn3PLyyJk1KZVR-qswx-sHYwT5qorHdH2yDozhyJPeZ9aGtodyHCA_HwmrRur3OHhnKs_xHgWlwJLsJcTxFS3oQkoFOKFMcVKrIPCnnk3giLzCE9MjvJsd8A/s640/2.png" width="640" /></a></div>
<br />
<br /></div>
<div>
<div style="border-width: 100%; direction: ltr;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 9.7777in;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 9.7777in;">
</div>
</div>
</div>
</div>
<div>
I decided to then give it 4GB memory and 2 CPU cores</div>
<div>
<div style="border-width: 100%; direction: ltr;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 9.7777in;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 9.7777in;">
</div>
</div>
</div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLgTc8ImgNe-9FnHbWJLLhkM-j8FpG5z-E_DmvFrvHuGFugEGcD9MpcAh9Ocw4cWAiKZMuEYNXC2vLBM6QmIX4n0z_5DLnWC4okcZBK04gO4tFUU7recNdsYUmoVi9qs31nViesg/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="704" height="484" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLgTc8ImgNe-9FnHbWJLLhkM-j8FpG5z-E_DmvFrvHuGFugEGcD9MpcAh9Ocw4cWAiKZMuEYNXC2vLBM6QmIX4n0z_5DLnWC4okcZBK04gO4tFUU7recNdsYUmoVi9qs31nViesg/s640/3.png" width="640" /></a></div>
<br />
Chose to create a 40GB OS disk<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZTmvZket3-XpsEGmyg9LcjW1uOs1ANQE3rJ8oBQspNIfraij8q_Y7pT4Qd-GHDadUlY2je-SksrqX7bYkTyz0u0jrl0uIo0s7luEgQRRPtSsd8x9HwQU9cde0gumFrtvaw8-8Dw/s1600/4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="704" height="484" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZTmvZket3-XpsEGmyg9LcjW1uOs1ANQE3rJ8oBQspNIfraij8q_Y7pT4Qd-GHDadUlY2je-SksrqX7bYkTyz0u0jrl0uIo0s7luEgQRRPtSsd8x9HwQU9cde0gumFrtvaw8-8Dw/s640/4.png" width="640" /></a></div>
<br /><br />
Then install from bootable CD and pointed the location of the image file to the downloaded ISO image<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkUGp-TVKHzbTt4_OzYObopdZDs1KlwiLDo5X3b20i3RzF-GgL5OkiDoLBPdwo70bSqYgC_jSCSU8M8pn_nYqLzMOVNuUZ5c7mdmqGJhErdvTkMrutZoOw0NXCEuThr67i-iS7OA/s1600/5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="704" height="484" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkUGp-TVKHzbTt4_OzYObopdZDs1KlwiLDo5X3b20i3RzF-GgL5OkiDoLBPdwo70bSqYgC_jSCSU8M8pn_nYqLzMOVNuUZ5c7mdmqGJhErdvTkMrutZoOw0NXCEuThr67i-iS7OA/s640/5.png" width="640" /></a></div>
<br />
Click through next to end and finish the creation wizard. Then right click on the newly created VM and choose "Connect".<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRDvo8rScaPjCiV02YY_SJsT6XMeoZG30E6NhyphenhyphenM5u8kS5GhXgV2B9lKgtCrZXG4qttLLmk9r6SENeQ6iM2qrco-T0ftbMK2FMXat-VrF0OlkhOWGukzDiuDmXKOuh9jzT1ckqlGQ/s1600/6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="871" data-original-width="1560" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRDvo8rScaPjCiV02YY_SJsT6XMeoZG30E6NhyphenhyphenM5u8kS5GhXgV2B9lKgtCrZXG4qttLLmk9r6SENeQ6iM2qrco-T0ftbMK2FMXat-VrF0OlkhOWGukzDiuDmXKOuh9jzT1ckqlGQ/s640/6.png" width="640" /></a></div>
<br />
<h2 style="text-align: left;">
Install Windows 7</h2>
<br />
At this point if everything went well the VM has booted off the installation ISO and we are on the following screen. Choose "Clean install" and proceed through the installation wizard.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZk7i9C7jaxbv_0oNrLaHLI4Papd4fIUolRUuRbXQRwqyx1Pn1a79zdhY-EmIFWCh3il4-APNZJt4-P_gu_ZV4YD4HzayBNe1ZUXWspJPnoVPBwHPSrYB1w4rIVuLptj0qMmewag/s1600/7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="704" data-original-width="802" height="560" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZk7i9C7jaxbv_0oNrLaHLI4Papd4fIUolRUuRbXQRwqyx1Pn1a79zdhY-EmIFWCh3il4-APNZJt4-P_gu_ZV4YD4HzayBNe1ZUXWspJPnoVPBwHPSrYB1w4rIVuLptj0qMmewag/s640/7.png" width="640" /></a></div>
<br /><br />
Finally installation starts.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd3z66dumk3it9Qka3Sgj-rJX1gsmWBL-0HOjBZ5RfxMiLuyoffsS_5s4pfN9Gowxbjr6M1wELcwx0phZwKzCS0MdywvQ4PwZBjOyuIPJngAzcFR8h20v3-iR-3a-99zDWzx-99Q/s1600/8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="704" data-original-width="802" height="560" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd3z66dumk3it9Qka3Sgj-rJX1gsmWBL-0HOjBZ5RfxMiLuyoffsS_5s4pfN9Gowxbjr6M1wELcwx0phZwKzCS0MdywvQ4PwZBjOyuIPJngAzcFR8h20v3-iR-3a-99zDWzx-99Q/s640/8.png" width="640" /></a></div>
<div class="" style="clear: both; text-align: left;">
<br /></div>
<div class="" style="clear: both; text-align: left;">
A reboot later we have Windows 7 starting up!</div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLb77wTIbqGwP-SGRu778OfPHRUGnYUjrC5bP4jKbv9g01Or_VaI_sG_UhS70KH5w34WHu59BZ4vTKd6xlLzfhH8tlLNq5pTWND4RjWvt7C9l7H4rv4XYaaPXLzmPFN0RP8YTPAQ/s1600/9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1026" height="542" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLb77wTIbqGwP-SGRu778OfPHRUGnYUjrC5bP4jKbv9g01Or_VaI_sG_UhS70KH5w34WHu59BZ4vTKd6xlLzfhH8tlLNq5pTWND4RjWvt7C9l7H4rv4XYaaPXLzmPFN0RP8YTPAQ/s640/9.png" width="640" /></a></div>
<br />
Created a username, password<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfkp8hq5lNw-gwOGDDmc0Pv-hNedbzLfwNFpL1SRubIbebxf3LuLRzLDRLCs1GfaDl8_WGaS6CrPtlmknDdVFNAsDOYE78KT3esqBlOqAuexMixkb0-U5Ub61KwA00E_L_esiCnQ/s1600/10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1026" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfkp8hq5lNw-gwOGDDmc0Pv-hNedbzLfwNFpL1SRubIbebxf3LuLRzLDRLCs1GfaDl8_WGaS6CrPtlmknDdVFNAsDOYE78KT3esqBlOqAuexMixkb0-U5Ub61KwA00E_L_esiCnQ/s640/10.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Finally booted into Windows 7 and here's my website displayed in Internet Explorer</div>
<div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaVQjKNZKFLIx6ZztXgE_8NQgN9ncudZc51tLJtB52GIk10JRQ3uHblI-om0asIzh7TG9i1Dp8qhebrCha8X2FWt4NmvYPBu97A2YE9risL9jd1vki2ElACpdqfLTUkri2JGlgew/s1600/11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1026" height="542" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaVQjKNZKFLIx6ZztXgE_8NQgN9ncudZc51tLJtB52GIk10JRQ3uHblI-om0asIzh7TG9i1Dp8qhebrCha8X2FWt4NmvYPBu97A2YE9risL9jd1vki2ElACpdqfLTUkri2JGlgew/s640/11.png" width="640" /></a></div>
<br />
<br />
<h2 style="text-align: left;">
Secure by Checkpoint</h2>
<div>
Even though we have booted into Windows 7 soon this will be a totally unsupported OS, that means no security updates. This is a dangerous system to keep open to the internet. I recommend never doing that!! Also to be doubly sure, we will create a checkpoint. What that does is it creates a snapshot of the memory and the disk. So in case something malicious lands in this VM, we can go back to the pristine state when the snapshot was created and hence rollback any changes made by the virus or malware.</div>
<div>
<br /></div>
<div>
To create a checkpoint right click on the VM in hyper-v manager and choose Checkpoint</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgybmJOk7iu_xzL3Dkl6ft7Dqhb-oPGaO6Mf5CUOpQ5sjtvO3VmOOY4fAUgvIUoQ7XgZxE3HPdDgEXKnkoDH8Q9DXvVrn-qPJW2wK1K35AJW5ikvydKavBlAnhhxH4K3NBgOV6VgQ/s1600/12.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="743" data-original-width="782" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgybmJOk7iu_xzL3Dkl6ft7Dqhb-oPGaO6Mf5CUOpQ5sjtvO3VmOOY4fAUgvIUoQ7XgZxE3HPdDgEXKnkoDH8Q9DXvVrn-qPJW2wK1K35AJW5ikvydKavBlAnhhxH4K3NBgOV6VgQ/s640/12.png" width="640" /></a></div>
<div>
<br /></div>
<div>
You can see the checkpoints created in the Hyper-V Manager.</div>
<div>
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixLPKWjZFWIFEmYTQZ64mhlPtGGYzGTYvLpvegCbSdD8eW55AEJmKXxmRliuyerZ7M-t24ZP6LqSUJZdbBPj1Ow9aDkHLhziiNNLPe0tBkztQu9ATbhiYjjt-UbL-swGDZdcvDMw/s1600/13.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="743" data-original-width="782" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixLPKWjZFWIFEmYTQZ64mhlPtGGYzGTYvLpvegCbSdD8eW55AEJmKXxmRliuyerZ7M-t24ZP6LqSUJZdbBPj1Ow9aDkHLhziiNNLPe0tBkztQu9ATbhiYjjt-UbL-swGDZdcvDMw/s640/13.png" width="640" /></a></div>
<br />
<div class="" style="clear: both; text-align: left;">
Lets make a change to the Windows 7 VM by creating a file named "Howdy I am created.txt" on the desktop.</div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFhTGwe4aQjiKaP0BPt58292nZ594g1zaUSRRBy3NB-lHNaY3rwX7EZ9ayxg8s-kfvr0si4zoptapGK5C5dXPjX4fmMD5JQrnAdaahamOdR0wyDvlMd9HdSegGLy0iGIzdZPncFQ/s1600/14.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1026" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFhTGwe4aQjiKaP0BPt58292nZ594g1zaUSRRBy3NB-lHNaY3rwX7EZ9ayxg8s-kfvr0si4zoptapGK5C5dXPjX4fmMD5JQrnAdaahamOdR0wyDvlMd9HdSegGLy0iGIzdZPncFQ/s640/14.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="" style="clear: both; text-align: left;">
Since the checkpoint was created before creating the file, I can revert back to the checkpoint by right-click on the checkpoint and choosing Apply.</div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimQdQa8jZ0Us15kp098Vq3U_mKrzFi2e97eh-dCE3xd2XYo_hIvS57VyXPhlD_60Di6s3D7tiId3iKy7aPlH92QMmMhmRPmWZWdVbZWZLfjrMES1qQMM-8WDqQRb_xNFL1RowmHg/s1600/15.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="743" data-original-width="782" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimQdQa8jZ0Us15kp098Vq3U_mKrzFi2e97eh-dCE3xd2XYo_hIvS57VyXPhlD_60Di6s3D7tiId3iKy7aPlH92QMmMhmRPmWZWdVbZWZLfjrMES1qQMM-8WDqQRb_xNFL1RowmHg/s640/15.png" width="640" /></a></div>
<br />
<a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=6434675" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a>After applying the checkpoint when I go back into the VM, the created file is all gone!!!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh48PAwZ79_P0WJ7pprUxHrUpH_9bFQISTGRlnmRdat_OCY4j9q6SE4-FNXa1D0FDCKcKVpkf0O_o06-GgmkRkZAPPo8Mt7b_vSWWzNIYcIqXlGWshLu0R4pEYqLOzVLR3EtYP6uQ/s1600/16.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1026" height="542" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh48PAwZ79_P0WJ7pprUxHrUpH_9bFQISTGRlnmRdat_OCY4j9q6SE4-FNXa1D0FDCKcKVpkf0O_o06-GgmkRkZAPPo8Mt7b_vSWWzNIYcIqXlGWshLu0R4pEYqLOzVLR3EtYP6uQ/s640/16.png" width="640" /></a></div>
<br />
<h2 style="text-align: left;">
Finally</h2>
</div>
<div>
This is a hack at best and not recommended. However, if for some applications or other need where you "have" to run Windows 7, this can be an option.</div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-27614511375231453852020-01-06T01:09:00.001-08:002020-02-24T12:38:13.829-08:00Chobi - A static photo gallery generator<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKw0kS0kzxCsdmWz6qI-riAHiTqw3XiwGZDiDW-y6phZsfoOLHoEIy2m2uH2ZVRF5q5AhxGXCk1rqd0QxucRACyCj137UdBaNo0z5DUpAKsa_STIpUxHDrTPLcdhtaAa8p9v-t1A/s1600/IMG_5529-Pano-Edit-Edit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="759" data-original-width="1600" height="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKw0kS0kzxCsdmWz6qI-riAHiTqw3XiwGZDiDW-y6phZsfoOLHoEIy2m2uH2ZVRF5q5AhxGXCk1rqd0QxucRACyCj137UdBaNo0z5DUpAKsa_STIpUxHDrTPLcdhtaAa8p9v-t1A/s640/IMG_5529-Pano-Edit-Edit.jpg" width="640" /></a></div>
<br />
<br />
I love using <a href="https://todo.microsoft.com/tasks/en-us/">Microsoft Todo</a> and before taking time off in December I create a holiday todo list. I tend to be at home with the family and do bunch of projects around. I try to ensure that I am not doing only work related projects during that time, so put in a ceiling of half a week for coding related stuff. Other Todos generally involves carpentry, DIY home projects, yardwork, cleaning etc.<br />
<br />
One of the projects was to update my online photo gallery. Now being a programmer I made it way more complicated than I should've. I decided to code up a minimalistic program to generate static photogallery out of folders of images I export out of Adobe Lightroom. As I mentioned above one of the requirement was to finish it in around 3 days.<br />
<br />
I am happy to share that I have the project done and the sources are available at <a href="https://github.com/abhinababasu/chobi">https://github.com/abhinababasu/chobi</a>. It took me about 3 days and most of the time was spent figuring out UI stuff which I rarely do and pondering about which photos to put in the gallery.<br />
<br />
The code is in go and it does the following<br />
<br />
<ol style="text-align: left;">
<li>It iterates through a folder of images (sub-dir not supported yet) and copies the images to a destination</li>
<li>Also places thumbnails (configurable size) into the destination</li>
<li>There is a template html that it modifies to display those images</li>
<li>It also uses some client side script to </li>
<ol>
<li>Randomize the image order</li>
<li>Show a carousel of the images</li>
<li>A thumbnail gallery at the bottom</li>
<li>Automated photo rotation</li>
</ol>
</ol>
<div>
Here's a screenshot of the sample landscape gallery.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHSjWCTt4GRzexWO_w-PVx5Lq8b6vCSpK73Ah0McZ8Okte62dMa1Fhdp1IfsepIUmz7ZjLG12X4Rvg8u5SjHd4Eq6tpBv6BURCQuebk0oSELnpnttZlDlYoIfZjFHWWyVwiZdnVw/s1600/Capture.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1008" data-original-width="1600" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHSjWCTt4GRzexWO_w-PVx5Lq8b6vCSpK73Ah0McZ8Okte62dMa1Fhdp1IfsepIUmz7ZjLG12X4Rvg8u5SjHd4Eq6tpBv6BURCQuebk0oSELnpnttZlDlYoIfZjFHWWyVwiZdnVw/s640/Capture.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
Since this was very time-bound project there are tons to stuff left to do, some basic bugs abound as well. But I decided to timeout on the effort for now and revisit again hopefully in the spring.</div>
</div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-73986246846566135992019-09-29T14:22:00.004-07:002020-10-08T16:45:36.318-07:00Azure Dedicated<div dir="ltr" style="text-align: left;" trbidi="on">
I remember a discussion with a group of friends around 8 years back. Microsoft was in it’s early days of becoming a leader in the cloud. Those friends, all techies in the Seattle area had varying expectation on how it would work out. Many thought that a full blown move was few decades away because their experience indicated that all big companies ran on stack that was very old and simply couldn’t be moved to the cloud any time soon.<br />
<br />
Waves of workloads have been since moving to the cloud. A new brew of startups were cloud native from the start and they were the first to use the power of cloud. Many large and small enterprises had already virtualized workloads and they moved as well. Some moved their new workloads (green-field), some even followed lift-n-shift with some modifications (brown-field) into the cloud.<br />
<br />
However, a class of large enterprises were stuck in their data centers. They wanted to use the power of the cloud, they wanted to use IoT integration, Machine-learning and the capability of elastic growth of their applications, but the center of their systems were running on some stack that did not run in the standard virtualization offered in the cloud. These enterprises said that if they cannot move those workloads into the cloud, they would need to keep the lights on in their data-centers and moving some peripheral workloads simply did not make sense.<br />
<br />
This is where Azure Dedicated and we come into the picture.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMVDMP2kGwcdZKtI2p-Pbxam8DASCynXtPwGKbLPthhjq36Zxgn4dERhp4d6lTQAGxzePUxJVDupueddOWgK0OeNNxKQ_afmWhJ2q-qjj4CfVWpZuCtvNHscLopMHODoBb7BaJ3w/s1600/Azure+Dedicated.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMVDMP2kGwcdZKtI2p-Pbxam8DASCynXtPwGKbLPthhjq36Zxgn4dERhp4d6lTQAGxzePUxJVDupueddOWgK0OeNNxKQ_afmWhJ2q-qjj4CfVWpZuCtvNHscLopMHODoBb7BaJ3w/s1600/Azure+Dedicated.png" width="600" /></a></div>
<h3 style="text-align: left;">
<strong>SAP HANA Large Instance</strong></h3>
For some of these customers that #$%#@ is SAP HANA in-memory DB on a single machine with 768 vCPUs and 24 terabytes of ram (yup) and we have them covered. Some wanted to scale those out to 60 terabytes in memory, we have them covered too with our bare-metal machines running in Azure. See <a href="https://blog.bonggeek.com/2018/10/hana-large-instances-on-azure.html">SAP HANA Large Instances on Azure</a>.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /></div><div dir="ltr" style="text-align: left;" trbidi="on">We started our journey in this area with this workload. Now we have evolved into our own little organization in Azure called Azure Dedicated and also support the following workloads.<br />
<h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">
Azure VMware Solutions</h3><div dir="ltr" style="text-align: left;" trbidi="on">Some customers wanted to run their VMware workloads and we have <a href="https://azure.microsoft.com/en-us/services/azure-vmware/">Azure VMware Solution</a> for them.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /></div><h3 style="text-align: left;">
<strong>Hardware Security Modules</strong></h3>
In partnership with other teams in Azure we support <a href="https://azure.microsoft.com/en-us/services/azure-dedicated-hsm/">HSM</a>, which are standard cryptographic appliances powering say financial institutions.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /><h3 style="text-align: left;">
<strong>Cray Supercomputer?</strong></h3>
So you need to simulate something or do ML on tens of thousands of cores, we have <a href="https://azure.microsoft.com/en-us/solutions/high-performance-computing/cray/">Cray super computers running inside Azure</a> for that!!</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /><h3 style="text-align: left;">
<strong>Azure NetApp Files</strong></h3>
Working closely with the storage team we deliver demanding file based workloads running on <a href="https://azure.microsoft.com/en-us/services/netapp/">Azure NetApp files</a></div><div dir="ltr" style="text-align: left;" trbidi="on"><br /><h3 style="text-align: left;">
SkyTap</h3>
In partnership with <a href="https://www.skytap.com/azure/">SkyTap </a>we provide IBM Power workloads on Azure to customers.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /><h3 style="text-align: left;">
What next?</h3>As an organization we are always on the lookout for more exciting opportunities to bring the power of cloud to our customers, wherever they are. So keep looking out for new announcements from our org.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br /></div><h3 style="text-align: left;">How do we support these workloads?</h3><div>Our team builds cutting edge, highly available, scalable, distributed systems to power these workloads in Azure. Our engineering stack uses linux based containers on Kubernetes, AKS, Helm, linkerd, fluentd, docker, go-lang, python, .netcore, C#, F# etc. Some samples of how we build and run our systems</div><div><ol style="text-align: left;"><li><a href="https://blog.bonggeek.com/2020/04/managing-baremetal-blades-in-azure.html">Managing BareMetal extreme high memory blades in Azure</a></li><li><a href="https://blog.bonggeek.com/2020/06/building-azure-monitor-for-sap-solutions.html">Monitoring HANA Databases</a></li></ol></div><div><br /></div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com2tag:blogger.com,1999:blog-6434675.post-83105629219391250702018-10-09T19:00:00.000-07:002018-10-09T20:32:46.987-07:00SAP HANA Large Instances on Azure<p><a href="https://lh3.googleusercontent.com/-G3mBXrVatLI/W71rGjb2tHI/AAAAAAAAOkY/ctuX6QXWVmUhJBNRM_3Lr9NlannSNCLdwCHMYCw/s1600-h/image%255B9%255D"><img width="558" height="293" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH7YIWYoL59PBoOsiAil_6IAvdHOWQqJDWsfjqqq0fLJmeMDuv12S_nlFn3ZGGuIDvHhbsgLX1a8ZhzbeXDTfWFJJ0cZ47rsyuPAkw7LbhKPSkHE772d6mXd3foDruSpNVXPvuYg/?imgmax=800" border="0"></a></p><p>Over the past year I have been working to light up bare-metal machines on Azure Cloud. These are specialized bare-metal machines that have extremely high amount of RAM and CPU and in this particular case, purpose built to run <a href="https://www.sap.com/products/hana.html">SAP HANA</a> in-memory database. We call them the <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-overview-architecture">HANA Large Instance</a> and they come certified by SAP (see list <a href="https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/iaas.html#categories=Microsoft%20Azure">here</a>).</p><p>So why bare-metal? They are huge high performance machines that goes all the way up to 24TB RAM (yup) and 960 CPU threads. They are purpose built for HANA in memory database and have the right CPU/Memory ratio and high performance storage to run demanding OLTP + OLAP workloads. Imagine a bank being able to load every credit card transaction in the past 5 year and be able to do analytics including fraud detection on a new transaction in a few seconds, or track the flow of commodities from the worlds largest warehouses to millions of stores and 100s of millions of customers. These machines come with 99.99% SLA and can be reserved by customers across the world in US-East, US-West, Japan-East, Japan-West, Europe-West, Europe-North, Australia-SouthEast, Australia-East to SAP HANA workloads.</p><p>In SAP TechEd and SAPPHIRE I demoed bare-metal HLI machines with standard <a href="http://portal.azure.com">Azure Portal</a> integration. Right now customers can see their HLI machines in the portal and coming soon even reboot them from the portal.</p>
<h2><strong>Portal preview</strong></h2><p>Click on the screenshot below to see a recorded video on how the Hana Large Instances are visible on the Azure portal and also how customers can raise support requests from the portal.</p>
<p><a href="https://1drv.ms/v/s!AjXx7atxODBCjOZXrInieItIKwsDeg"><img width="644" height="366" title="image" style="display: inline; background-image: none;" alt="Portal screenshot" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAgXlUf6swFCi7d2edyP1OgcAdQVMjIRwPKB2ffLFaneonQK4IMBsvH2bd-vDlnVnSrcqlM1nz8w9KPeoZyEm5C1KZWa8PDMI5CWt69wZ06MgmIZSM_2aDnTzUt85sBXbEnYKb9w/?imgmax=800" border="0"></a></p>
<p>
<strong>Reboot Demo</strong></p><p>This is something we are working on right now and will be available soon. Click on the screenshot below to see the video of a HANA Large instance being rebooted from the portal directly.<a href="https://1drv.ms/v/s!AjXx7atxODBCjOZWI9A4c3zyLlcKLA"><img width="644" height="441" title="image" style="display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQkW3Xs4lcJ6ABOGoj18btcg5_41rZpR9J0c03dgMEQLD4n-fysFuU1wCIakOMIy6qK2cdzafX9cU9BL0jNegFHaW-UN235b4yIn-48cBV-NKM6TGccdUcoIxwrmCpUVhlSDEYfQ/?imgmax=800" border="0"></a></p>
<h2>Getting Access</h2><p>Customers with HLI blades can run the following CLI command to register our HANA Resource Provider</p><pre>az provider register --namespace Microsoft.HanaOnAzure</pre><p>Or alternatively using the <a href="http://portal.azure.com">http://portal.azure.com</a>. Go to your subscription that has HANA Large Instances, select “Resource Providers”, type “Hana” in the search box. Click on register.<p><a href="https://lh3.googleusercontent.com/-WIy8VZS4WSA/W71rHzevPWI/AAAAAAAAOkg/7E5AJdhHOXEvJnEg5b9qRrAiZFUaCjGOwCHMYCw/s1600-h/image%255B20%255D"><img width="640" height="225" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNgaND_rzFQsPjI-TL8Lxroa6zEIgMHQ4po37wlY62CZnt7hN65R8V2_CWpF2OghaWUbQ9VDDH50ougGB7QNca2Q6piI3_w7k5zM06N8q2psHnE0WCzwNuswUqv9-7YcJVMOwexQ/?imgmax=800" border="0"></a><h2>Questions?</h2><p>Send them to <a href="mailto:sap-hana@microsoft.com">sap-hana@microsoft.com</a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-70387118347575055942018-05-31T19:38:00.001-07:002018-06-13T13:30:45.993-07:00Deploy Cloud Dev Box on Azure with Terraform<p><a href="https://lh3.googleusercontent.com/-V3g6qEBe200/WxCxhgf_GvI/AAAAAAAAL_E/WA3RyRQuKjwfQ2Zdmswpbm2Y91Np9d_EgCHMYCw/s1600-h/image%255B12%255D"><img width="552" height="310" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfjAXDM4NI-cEmtolKimL_fNQ5QY6SLW1G14e-diCRWnOfHMzrYUV9TSZ-fCNLbcC6UdZuizqMU7h32WrOB-gea6ij4fq1zhmMIymLOVG5yhVbHrMuG-zG8rdGOu3NZaEY5e5DCQ/?imgmax=800" border="0"></a></p><p><strong>Summary:</strong> See <a title="https://github.com/bonggeek/share/tree/master/cloudbox" href="https://github.com/abhinababasu/cloudbox">https://github.com/abhinababasu/cloudbox</a> for a terraform based solution to deploy VMs in Azure with full remote desktop access.</p><p>Now the longer form :). I have <a href="http://blog.bonggeek.com/2017/10/remote-ubuntu-desktop-on-azure.html">blogged in the past</a> about how to setup a Ubuntu desktop on Azure that you can RDP (remote desktop) into. Over the past few months I have moved onto doing most of my development work exclusively on cloud VM and I love having full desktop experience on my customized “Cloud Dev box”. I RDP into it from my dev box at work, Surface Pro, secure laptop etc.</p><p>I wanted to ensure that I can treat the <a href="https://devops.stackexchange.com/questions/653/what-is-the-definition-of-cattle-not-pets">box as cattle and not pet</a>. So I came up with a terraform based scripts to bring up these cloud dev boxes. I have also shared them with my team in Microsoft and few devs are already using it. I hope it will be useful to you as well incase you want something like that. All code is at <a href="https://github.com/abhinababasu/cloudbox">https://github.com/abhinababasu/cloudbox</a> </p><p>A few things about the main terraform script at <a title="https://github.com/abhinababasu/cloudbox/blob/master/cloudVM.tf" href="https://github.com/abhinababasu/cloudbox/blob/master/cloudVM.tf">https://github.com/abhinababasu/cloudbox/blob/master/cloudVM.tf</a> </p><ol><li>It is a good security practice is to ensure that your VM is locked down. I use Azure NSG rules to ensure that the VM denies in-bound traffic from Internet. I accept parameters to the script where you can give IP ranges which will then be opened up. This ensures that your VM is accessible from only safe locations, in my case those are IP ranges of Microsoft (from work) and my home IP address.</li><li>While you can use just the TF file and setup script I have a driver script at <a title="https://github.com/abhinababasu/cloudbox/blob/master/cloudshelldeploy.sh" href="https://github.com/abhinababasu/cloudbox/blob/master/cloudshelldeploy.sh">https://github.com/abhinababasu/cloudbox/blob/master/cloudshelldeploy.sh</a> that you might find useful</li><li>Once the VM is created I use remote execution feature of terraform to run the script in <a title="https://github.com/abhinababasu/cloudbox/blob/master/cloudVMsetup.sh" href="https://github.com/abhinababasu/cloudbox/blob/master/cloudVMsetup.sh">https://github.com/abhinababasu/cloudbox/blob/master/cloudVMsetup.sh</a> to install various software that I need including Ubuntu desktop and xrdp for remote desktop. This takes around 10 minutes atleast</li><li>By default Standard_F8s machine is used, but that can be overridden with larger sizes (eg. Standard_F16s). I have found machines smaller than that doesn’t provide adequate performance. <strong>Note: You will incur costs for running these biggish VMs</strong></li></ol><p><strong>Pre-requisite</strong></p><p>Obviously you need terraform installed. I think the whole system works really well if you launch from <a href="https://shell.azure.com">https://shell.azure.com</a> because that way all the credential stuff is automatically handled, and cloud shell comes pre-installed with terraform.</p><p>If you want to run from any other dev box, you can need to have Azure CLI and terraform installed (use <a href="https://github.com/abhinababasu/cloudbox/blob/master/installterraform.sh">installterraform.sh</a> script for it) . Then do the following where subsId is the subscriptionId under which you want the VM to run.</p>
<pre>az login
az account set --subscription="<some subscription Id>"</pre>
<p>While you can download the files from <a href="https://github.com/abhinababasu/cloudbox">here</a> and use it, you should be better of by customizing the cloudshelldeploy.sh script and then running it. I use the following to run</p>
<pre>curl -O https://raw.githubusercontent.com/bonggeek/share/master/cloudbox/cloudshelldeploy.sh
chmod +x cloudshelldeploy.sh
./cloudshelldeploy.sh abhinab <password></pre><pre><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVuIyZgoTm4tBusCFLcfkWB__8L07uOS6tYgQxji9XJia3bGrpvp6RnMtKJC9GJ3ILG_BuPg1UbNF7s1bd8GnFiLhhX449msglnPTkzCRob2rQA106FRtrvyPb3WvR-DXKbkUs3Q/s1600-h/image%255B6%255D"><img width="575" height="326" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3MfVBnVCyyFr2YTk1CA1WvFFHS5GmVSnLHdREIoh4YTnSq7LLA1fDSH7xbVnDcObfUGWDUorBBTANW9nxCWROBGkEMs25gJ271pL-ZIBp4ZAfv1REh1vB9KDfQjZi0gFIEcoHvA/?imgmax=800" border="0"></a></pre>
<p>Finally </p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOsi9tM6uRBDKGRDcyvEDaTDf2JR_qVlD5L0x6N01E3q2NhyfeHWRxtgT1dps7HLxDnccwxcyoVO_SZxrWwFKXTzKy-lm5LuOo45iKTkCaP1aWTOM51D3_IkSay_dEEgcNMoWGtQ/s1600-h/image%255B17%255D"><img width="524" height="96" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5ajp7pDVvuo3t1aUBBoRFPhzrHIicZwUWtJUe0O7avXwt4PO-Q_OzxBkHm4wouM2KA-T6kC0MKEq0yjsnTdXsGkYKjY5s6JwfzxWKJdN60U7a8Fllew6sktFhEHOIXNmm791Xaw/?imgmax=800" border="0"></a></p><p>Now you can use a rdp client like mstsc to loginto the machine. </p><p><b>NOTE: In my experience 1080p resolution works well</b>, 4K lags too much to be useful. Since mstsc default is full-screen be careful if you are working on hi-res display and explicitly use 1080p resolution.<p>There I am logged into my cloud VM.</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg07hRRmeqsF2QprKNymK_FTRVceCPTlbbmM52CdQEE1_kNZr5oRrQgwi7P7hB4Rbz43ur9UHbj_pXZphhOh60jtwusTK3dPqBQEiIAZ0F8SKsKMTcPS1TkwibLT7_OeYVxjGYSAA/s1600-h/image%255B23%255D"><img width="537" height="363" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-isbOSVdum_c/WxCxlchdOHI/AAAAAAAAL_g/i2gQSN67jGc4wzdeJHzeNToeii_kVfGIQCHMYCw/image_thumb%255B15%255D?imgmax=800" border="0"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-3372008789801581612018-05-15T17:26:00.001-07:002018-05-15T17:26:55.599-07:00Getting Azure Cloud Location<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_h8sTq5CJ_iZyhh00Sd71VnXwC6F8Yh1ddEKynQNvn2XScsVFnGa6gMGoLHaJrZz8OMxrvNgWat_93HFOHRH963KdpAjf_OoFtYncBvgC4odcwr-dnBmc-v2dSwLZqOXZzSUNfQ/s1600-h/image%255B8%255D"><img width="543" height="351" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-G_0r3ziJ_xQ/Wvt6yw6IibI/AAAAAAAALvg/USjcE_GcipoT7JItNdDJuSeHKX1iuqWGQCHMYCw/image_thumb%255B4%255D?imgmax=800" border="0"></a></p><p>I have had got some ask on how to discover which Azure cloud the current system is running on. Basically you want to figure out if you are running something in the Azure public cloud or in one of the specialized government clouds.</p><p>Unfortunately this is not currently available in <a href="https://aka.ms/azureimds">Instance Metadata Service</a>. However, it can be found out using a an additional call. The basic logic is to get the current location over IMDS and then call Azure Management API to see which cloud that location is present in.</p><p>Sample script can be found at <a title="https://github.com/bonggeek/share/blob/master/azlocation.sh" href="https://github.com/bonggeek/share/blob/master/azlocation.sh">https://github.com/bonggeek/share/blob/master/azlocation.sh</a></p>
<pre>#!/bin/bash
locations=`curl -s -H Metadata:True "http://169.254.169.254/metadata/instance/compute/location?format=text&api-version=2017-04-02"`
# Test regions
#locations="indiasouth"
#locations="usgovsouthcentral"
#locations="chinaeast"
#locations="germanaycentral"
endpoints=`curl -s https://management.azure.com/metadata/endpoints?api-version=2017-12-01`
publicLocations=`echo $endpoints | jq .cloudEndpoint.public.locations[]`
if grep -q $locations <<< $publicLocations; then
echo "PUBLIC"
exit 1
fi
chinaLocations=`echo $endpoints | jq .cloudEndpoint.chinaCloud.locations[]`
if grep -q $locations <<< $chinaLocations; then
echo "CHINA"
exit 2
fi
usGovLocations=`echo $endpoints | jq .cloudEndpoint.usGovCloud.locations[]`
if grep -q $locations <<< $usGovLocations; then
echo "US GOV"
exit 3
fi
germanLocations=`echo $endpoints | jq .cloudEndpoint.germanCloud.locations[]`
if grep -q $locations <<< $germanLocations; then
echo "GERMAN"
exit 4
fi
echo "Unknown'
exit 0
</pre>
<p>This is what I see for my VM</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLxCKaDD8CyYwTGNZgvQwkpHJqIDGwPYauLkZVqPzmmzz4RExhwQJvnDwU4xQl5_waLgHi26k_ipZBBnGgtVUhzjVW535eMfhA76sjpXKRYdqyUkEqINy5RSyId6r72KCU8M1BIA/s1600-h/image%255B3%255D"><img width="240" height="34" title="image" style="display: inline;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHDNsw3gJvmmF3b0aKS8qjK0Y_Ms19ai7cIBAL5vyR62kCUbuhzjyQd3UOZdapXVRp3X71NJRgphDTWbmyynYqlD3-J4nVdcnxy7RQtNK13pHZX1yDhiD5itL_fDAWITaYXk60rw/?imgmax=800"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-64349781940104156042018-03-26T10:06:00.001-07:002018-03-26T10:12:47.005-07:00Azure Serial Console<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvYniN5AgQg9sPfVXKGiNOpnS61JBY-2fhuYFD6FlY1SFP6pqWNpWuKWYYY2nMTzxFRfC7_RbAK05n6aXRSSP6hyfXSKJ7tKHC1zD9zp8z5SGi5qESd4cnGVYAq41d8NkvvxSbmg/s1600-h/28452510890_0b229a6726_z%255B4%255D"><img width="512" height="341" title="28452510890_0b229a6726_z" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="28452510890_0b229a6726_z" src="https://lh3.googleusercontent.com/-BIYcYxCZ1CE/Wrkoli0YajI/AAAAAAAAKsc/248HKD0vItMa1WW7OO2YfZ_iHHP5S_6YQCHMYCw/28452510890_0b229a6726_z_thumb%255B2%255D?imgmax=800" border="0"></a></p><p>My team just <a href="https://aka.ms/serialconsolepreviewblog">announced</a> the public preview of Azure Serial console. This has been a consistent ask from customers who want to recover VMs in the cloud. Go to your VM in <a href="http://portal.azure.com">http://portal.azure.com</a> and then click on the Serial Console button</p><p><a href="https://lh3.googleusercontent.com/-qdOYIh4LdrU/WrkqDDKjgQI/AAAAAAAAKtg/rc0qqA0OTNA74oJQhcsZSn2EWqzIl9yEACHMYCw/s1600-h/image%255B21%255D"><img width="455" height="370" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMJMNf_oZ9X7S6GjFmUMjXGHLs6lJHRAdfgsa1zLk43no3Z1txOyWRng3TVZhK6FDQw-kNd5lY1ytjqebORZA9_T0mheQ6fDvNibHtGD-ds3TiriYfHE3F638xZJIbuG5CTV6JmA/?imgmax=800" border="0"></a></p><p>This opens a direct serial console connection to your VM. It is not required to have the VM open to internet. This is amazing to diagnose VM issues. E.g. if you are not able to SSH to the VM for some reason (blocked port, bad config change, busted boot config). You drop into the serial console and interact with your machine. Cool or what!!</p><p><a href="https://lh3.googleusercontent.com/-IywbuusD7bI/WrkomlO_2tI/AAAAAAAAKso/JH9elANIvckmoZvsDKQZEhsl6BpaOrVwgCHMYCw/s1600-h/Capture2%255B5%255D"><img width="510" height="143" title="Capture2" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="Capture2" src="https://lh3.googleusercontent.com/-A5dz-E0fzbE/WrkonGA5hhI/AAAAAAAAKss/uLpcgHZxdSA2fGDP_rNhXWsvHyDYyi8vwCHMYCw/Capture2_thumb%255B3%255D?imgmax=800" border="0"></a></p><p><a href="https://lh3.googleusercontent.com/-9Y8ldJ8U4W0/WrkonUyCwYI/AAAAAAAAKsw/nw21tt38j-0vPzc9cx-OWAOgragB91qagCHMYCw/s1600-h/Capture3%255B4%255D"><img width="529" height="337" title="Capture3" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="Capture3" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-MI5rea5Hds2aS-O2XtG25H0f-ghGGD_bx3MXNHKlRx6Z7ZNKtkGQEoy22LgfF1U-DzpZI0m3hac24tg-Kp0YEu94mq_BWaL9AbOH4r2B78MFwW91lWK92tybmJ4N0FU88U3Vjg/?imgmax=800" border="0"></a></p><p>To show you the difference between a SSH connection and serial console, this is my machine booting up!!</p><p><a href="https://lh3.googleusercontent.com/-RVpq6aPQLww/WrkooZ0jvYI/AAAAAAAAKs4/HDxlaDgm4pscepubw1I1I45sI1C6DArlQCHMYCw/s1600-h/image%255B15%255D"><img width="526" height="241" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVQvZLbZIvI1qu4GA-puPF73vMIQxWwBoDFY1zHaJK4HH1AWLFcii5hv_bpCMDKxKuGQxsTjCIl1yCXGWUrCg20GQH3nKbYOfJD1wxNS_T7ktXbRSLaIpsWfbk3CJZukd1HQCJMw/?imgmax=800" border="0"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-29625750234638868332017-10-05T15:05:00.000-07:002017-10-05T19:25:23.529-07:00Remote Ubuntu desktop on Azure<p><a href="https://lh3.googleusercontent.com/-YQiAFlBER0g/WdasibOGOMI/AAAAAAAAH2k/_aXgPt3nFEwL2Mrf_6ekCA6P9zJ4p9p7wCHMYCw/s1600-h/image%255B22%255D"><img width="539" height="296" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-1yfFYGONiCk/Wdasi701WTI/AAAAAAAAH2o/tnNdfupFBrEZyg-Z0pc7_xstZgypRGqpACHMYCw/image_thumb%255B14%255D?imgmax=800" border="0"></a></p><p>For the past many months I have moved to have my dev boxes on the cloud. I am happily using a monster Windows VM and an utility Ubuntu desktop in the cloud. I realized after talking to a few people that they don’t realize how easy it is to setup Linux remote desktop in Azure cloud. Here goes the steps.</p><p>For VM configuration I needed a small sized VM with large enough network bandwidth to support remoting. Unfortunately on Azure you cannot choose networking bandwidth but rather all the VMs in a box gets networking bandwidth proportional to the number of cores they have. So I just created a VM based on the “Standard DS4 v2 Promo (8 vcpus, 28 GB memory)” and connected it to Microsoft ExpressRoute. If you are ok with public IP you skip setting the express route and ensure your VM has a public IP.</p><p>Then went to the Portal and enabled RDP. For that in the portal choose VM –> Networking and add rule to enable RDP.</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjenGGg8VBI24GqST1bfXIZwhdoyo_BMZJ6Y1aWlqMEiZsKCsxcRFu9zQVhf29EjdcKbGQHqt4gC4EtRJnu5VORHrThUNsC_1lRujQ4ilU91KYfb0UxJvXxkkUduaaETML9I-z7Ew/s1600-h/image%255B5%255D"><img width="200" height="296" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-IPwezYyszNI/WdasjZ16BJI/AAAAAAAAH2w/6QxZjyTo6sU22d3vQZWbg08LELZIAN1TACHMYCw/image_thumb%255B3%255D?imgmax=800" border="0"></a></p><p>Finally I sshed into my VM with</p><p>c:\bin\putty.exe <a href="mailto:abhinaba@AbhiUbuntu">abhinaba@AbhiUbuntu</a></p><p>Time to now install a bunch of tools</p>
<pre>sudo apt-get update
sudo apt-get install xrdp
sudo apt-get install ubuntu-desktop
sudo apt-get install xfce4
sudo apt-get update
</pre>
<p>Setup xsession</p>
<pre>echo xfce4-session > ~/.xsession
</pre>
<p>Open the following file</p>
<pre>sudo gvim /etc/xrdp/startwm.sh
</pre>
<p>Set the content to</p>
<pre>#!/bin/sh
if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG LANGUAGE
fi
startxfce4
</pre>
<p>start the xrdp service</p>
<pre>sudo /etc/init.d/xrdp start
</pre>
<p>And then from my windows machine mstsc /v:machine_ip. I am presented with the login screen</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjOn4jiJigTJEr-jCEMfCGlg5gVRvxlnG05lRqdaF4b0xGvx9aPALo8De66gKGPBEA_vK1LjjJt43XhLb-pF3MIL6GXkqjIJy6KvZs65nYBM9NFD3AQJgWYW0xxFjCXoTfCR8mKg/s1600-h/image%255B10%255D"><img width="575" height="332" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkST27LNBMTepozoqksYjqBQPyPLLH2AlfNf-W49M8x6o8YISmEfNhJyErsVMHm5tIAJIqhCci_bfJ6AT1qhhWcWOIy_xL8SH3CnZNRxAN9vfBj2kLH45CiPRVHVawN1vsUFNxlg/?imgmax=800" border="0"></a></p><p>then I have full Ubuntu desktop on Azure :)</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKcq_7Yh2J6-fQLwaP7giqJjMXNVcJvLKnqQpwrk04Jd37TCywhzoEnRzpXl3pkqEK_hoWd3ZDa1hjaUlqJWg2jSd95IoFVCwOh5TiNgeWmO6WA2RAYXq1vLk7wdnvZP2kU-cxcg/s1600-h/image%255B16%255D"><img width="586" height="330" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGlhw6MAlfAFS6PfrnnFfhroEKAZSr8c7dtGcbwkJCiubyg-dR4Lgafw7eTw3eqM0BKB6pHRRCsMhUmA6PdtxCP0r3bevQA6FJj7d_sKfxxMVbqozEjidIw3hQiHPqLir0j7dp0A/?imgmax=800" border="0"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-89430374787664627912017-09-13T16:02:00.000-07:002020-02-24T12:38:55.055-08:00Distributed Telemetry at Scale<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhthLAOk_u7x6ZDJODRh7raJt5qHQ792KSfcaUjWpQLl7zJM2yOvAZEvoOQ-xwZhM06IhfWWfJFyjA6BBN3MRfMYYP1ibmeZcEOGyifVsPHd7McBl2pC2j8X-i7FwU_S4lYLAiuMQ/s1600-h/image%255B22%255D" style="margin-left: 1em; margin-right: 1em;"><img alt="image" border="0" height="353" src="https://lh3.googleusercontent.com/-ffmK0XB3GK4/WdbBykKn2JI/AAAAAAAAH3U/O2XZVZyQfCAvNuswEevo66OH2LWCDZjUQCHMYCw/image_thumb%255B15%255D?imgmax=800" style="background-image: none; border: 0px currentcolor; display: inline;" title="image" width="640" /></a></div>
<br />
In <a href="http://blog.bonggeek.com/2017/09/designing-azure-metadata-service.html">Designing Azure Metadata Service</a> I elaborated on how we run Azure Instance Metadata Service (IMDS) at massive scale. Running at this scale in <a href="https://azure.microsoft.com/en-us/regions/">36 regions</a> (at the time of writing) of the world, on incredible number of machines is a hard problem to solve in terms of monitoring and collecting telemetry. Unlike other centralized services it is not as simple as connecting it to a single telemetry pipeline and get done with it.<br />
We need to ensure that<br />
<ol>
<li>We do not collect too much data (cost/latency)</li>
<li>We do not collect too less (hard to debug issues)</li>
<li>Data collection is fast</li>
<li>We are able to drill down into specific issues and areas of problem</li>
<li>Do all of the above when running in 36 regions of the world</li>
<li>Continue to do all of the above as Azure continues it’s phenomenal growth</li>
</ol>
To meet all the goals we take a three pronged approach. We break out telemetry to 3 paths<br />
<ol>
<li>Hot-path: Minimal numeric data that can be uploaded super fast (few second delayed) that we can use for monitoring our service and alert in case anomaly is detected</li>
<li>Warm-path: More richer textual data that are few minute delayed and we can use this to drill down into issues remotely in case hot-path flagged an issue</li>
<li>Cold-path: This gives us full fidelity data to monitor <br /></li>
</ol>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMQok2r6k6upELKb2owTFULh8bHRlVIuJXMB89og5q1QtfxePYt3kQkr4PuiVGnGoMW-SXAMcQ9iBPV3Smpk2uBjjUhgoJ6ELnzG9XBCMlyOnMS2pfmN-GzZJIfaggmlB0OB5F4w/s1600-h/image%255B10%255D"><img alt="image" border="0" height="325" src="https://lh3.googleusercontent.com/-Ps-pieUbU-0/Wbm5IAOJg0I/AAAAAAAAHeY/mIrDlizNDYMCWy2FgO3vcrtiGs6MtAkGwCHMYCw/image_thumb%255B6%255D?imgmax=800" style="background-image: none; display: block; float: none; margin-left: auto; margin-right: auto;" title="image" width="539" /></a><br />
<h2>
<strong><sub></sub>Hot-Path</strong></h2>
Even though we run on so many places we want to ensure that we have near real time alerting and monitoring and can quickly catch if something bad is happening. For that we use performance and functionality counters. These counters measure the type of response we are giving back, their latencies, data size etc. All of them are numeric and track each call in progress. We then have high speed uploaders in each machine with backends that can collect these. Then we attach these counters with alerts at per cluster level. We can catch latency issues, failures with few seconds delays. These counters only tell us if something is going bad and not why they are doing so. We have 10s of such numeric high speed telemetry coming from each IMDS instance.<br />
Here’s a snapshot of one such counter in our dashboard showing latency at 90th percentile. <br />
<a href="https://lh3.googleusercontent.com/-tChtPb-3dNw/WdLCR-hKf_I/AAAAAAAAH0Y/2H83F3GUMl0FSbTKDaPJzcI8HvmFq1FIgCHMYCw/s1600-h/image5%255B1%255D"><img alt="image" border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxR6PR44Pz4jviM4nK4Cl_N0w8imS2Pcs6Ul_kE5us9CtHv61Po5tBKW_Qx8-ClBtqdtl2ze2oDT85rTZy_bHa8A-yi-ts_USxHtaUA25tJpBiFrxtfB-y-sPRJFjm_Qx3NJvwTg/?imgmax=800" style="background-image: none; border: 0px currentcolor; display: inline;" title="image" width="543" /></a><br />
In addition we have external dial-tone services that keep pinging IMDS to ensure the services are up everywhere. If there is no response then likely there has been some crash or other deadlocks. We measure the dial-tone as part of our up-time and also have alerts tied to this.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtfK0NQTNU_-fepelddf5wCswVnjMuYQm_9vmfmtIjjkSjLnrhSGz9Fd3is0H_PBNjEC0Em6sSoKjJqCmYNWENhWS8uirgKgysgL3XNZGE2VpNq07NvcPlJa7XfGKAQkPE5j5YIg/s1600-h/image11"><img alt="image" border="0" height="259" src="https://lh3.googleusercontent.com/-AYwKQ4ABK3w/WdLCSgvrtFI/AAAAAAAAH0k/_QdQgLGuId0IokcP4z-ZWI2jzqcT_09NACHMYCw/image_thumb7?imgmax=800" style="background-image: none; border: 0px currentcolor; display: inline;" title="image" width="459" /></a><br />
<h2>
<sub></sub><span style="font-size: small;"><span style="font-weight: normal;">Warm-Path</span></span></h2>
If hot-path counter driven alerts tell us something has gone wrong and an on-call engineer is awaken, the next steps of business is to quickly figure out what’s going on. For that we use our warm-path pipeline. This pipeline uploads informational and error level logging. Due to volume the data is delayed by few minutes. The query granularity can also slow down fetching them. So one of the focus of the hot-path counters is that it can narrow down the location of problem to cluster level/machine level.<br />
The alert directly filters the logs being uploaded to a cluster/machine and brings up all logs. In most cases they are sufficient for us to detect issues. In case that doesn’t work we need to go into the detailed logs.<br />
<span style="font-size: small;"><strong>Cold-Path</strong></span><br />
Every line of logs (error/info/verbose) our service creates is stored locally on the machines with a certain retention policies. We have built tools so that given an alert an engineer can run a command from his dev box to fetch the log directly from that machine, wherever in the world the machine with the log exists. For hard to debug issues this is the last recourse.<br />
However, more cooler is that we use our <a href="https://azure.microsoft.com/en-us/services/cosmos-db/">CosmosDB</a> offering as a document store and store all error and info logs into that. This ensures the logs remain query-able for a long time (months) for reporting and analysis. We also run jobs that read the logs from these cosmos streams and then shove it into Kusto as structured data. Kusto is also available to users with the more fancier name of <a href="https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics">Azure Application Insights Analytics</a>. I was floored with the insight we can get with this pipeline. We upload close to 8 terabytes of log data a day into cosmos and still able to query all data over months in a few seconds<br />
Here’s a quick peek into seeing what kind of responses IMDS is handing out.<br />
<a href="https://lh3.googleusercontent.com/-VYio2FP5QGQ/WdbByynZSqI/AAAAAAAAH3Y/kfxa-ZiXAdA2RjLsQAEx946f6JbshjbnQCHMYCw/s1600-h/image%255B9%255D"><img alt="image" border="0" height="381" src="https://lh3.googleusercontent.com/-Hj3uHvGhBlU/WdbBzP_pkBI/AAAAAAAAH3c/XRBxxdrcqPI5cbKQth_oOI_m0Fyc2nZwwCHMYCw/image_thumb%255B6%255D?imgmax=800" style="background-image: none; display: inline;" title="image" width="383" /></a><br />
A look into the kinds of queries coming in.<br />
<a href="https://lh3.googleusercontent.com/-HrR3Gastirs/WdbBzWoamvI/AAAAAAAAH3g/ZmynxRhgVVAmwYrZHL6T57kZSsftUcNPACHMYCw/s1600-h/image%255B23%255D"><img alt="image" border="0" height="410" src="https://lh3.googleusercontent.com/-13NPBM6vh24/WdbBzlR_u4I/AAAAAAAAH3k/mhCiHPAAInYdFGCjqwSZvXwSz6SujHI0QCHMYCw/image_thumb%255B16%255D?imgmax=800" style="background-image: none; border: 0px currentcolor; display: inline;" title="image" width="580" /></a><br />
Distribution of IMDS version being asked for.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRCvgkw_hapBDMsVSG9fEG_tUOt2LgHzETQdPeafypJL5quq5Fa8rvfQ5ISGEgpqVaMx1B42xW8lzB_XG4O_QDqjxe381d2JxSRw8YGHMW8qXZGuatCuUsvre-ptC6-fvzwT0W0Q/s1600-h/image%255B29%255D"><img alt="image" border="0" height="425" src="https://lh3.googleusercontent.com/-QC0_q89Uwpo/WdbB0QP_fUI/AAAAAAAAH3s/dVvIjhUC8bgisgFcAAPo59WOh561HkgdQCHMYCw/image_thumb%255B20%255D?imgmax=800" style="background-image: none; display: inline;" title="image" width="441" /></a><br />
We can extract patterns from the logs, run regex matching and all sorts of cool filters and at the same time be able to render data across our fleet in seconds. </div>
Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-90858227114413724202017-09-11T10:15:00.001-07:002017-10-05T14:57:18.948-07:00Designing Azure Metadata Service<p><a href="https://lh3.googleusercontent.com/-q3gOLMEnyko/WdaquWcqraI/AAAAAAAAH2I/I4xz0ooqd_8YP9PZKIWg8GzpDVNfP5UqgCHMYCw/s1600-h/image25%255B2%255D"><img width="586" height="285" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYHExQd48MlUzbn3oeJyL8g36UiztN1LzmrCP_1hRVbrpbfn7thUC91_QSBubvDFJEMjclCajeXN2WZ-ryLb5qCdPaYVMsDiWnijdTfNrt_grDPnedX0AWkpgY6lh8zvcb4YIDew/?imgmax=800" border="0"></a></p><p>Some time back we have announced the general availability of <a href="https://aka.ms/azureimds">Azure Instance Metadata Service</a> (IMDS). IMDS has been designed to deliver instance metadata information into every IaaS virtual machines running on Azure over a REST endpoint. IMDS works as a data aggregation service and fetches data from various sources and surfaces it to the VM in a consistent manner. Some of the data can already be on the physical machine running the VM and others could be inside other regional service which are remote from the machine.</p><p>As you can imagine the scale of usage of this service is immense and spans across globe (at the time of writing 36 regions across the world) and Azure usage doubles YoY. So any design for IMDS has to be highly scalable and built for future growth.</p><p>We had many options to build this service both based on the various reliability parameters we wanted to hit as well as in terms of engineering ease.</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBh__esdzGecid43hMwG7RXEmS7ZRJ8Q9CGPrAgwUFfsfN_jq1HXOAMMBr1bLUomf09-7YpY_gM4JbYW0tY4XikElJiJ6fiIov6hkBACMTp9Im4AgyIkOoLld2PaEe6Z_h0vCFuA/s1600-h/image13%255B2%255D"><img width="529" height="291" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-qrgGNFYxVM0/WdaqvrCKRfI/AAAAAAAAH2U/TnbQGenBWucKBFqoMhsrJ5SOXwXoCymQgCHMYCw/image13_thumb%255B1%255D?imgmax=800" border="0"></a></p><p>Given a typical cloud hierarchical layout, you can imagine such a service to be built in any one of the following ways</p><ol><li>Build it like any other cloud service that runs on its own IaaS or PaaS infrastructure, with load-balancers, auto-scaling, mechanisms for distributing across regions, sharding etc.</li><li>Dedicate machines in clusters or data centers that run this service locally</li><li>Run micro-services directly in the physical machines that host the VMs</li></ol><p>Initially building a cross region managed service seems like a simpler choice. Pick up any of the standard REST stack, deploy using any of the many deployment models available in Azure and go with that. With auto-scaling, load balancers it should just work and scale. </p><p>Like with any distributed systems we looked into our CAP model.</p><ol><li><strong>C</strong>onsistency: We could live with a more relaxed eventual consistency model for metadata. You can update the metadata of a virtual machines by making changes to it in the portal or using Azure CLI and eventually the virtual machine gets this last updated value</li><li><strong>A</strong>vailability: The data needs to be highly available because various key pieces in the azure internal stack takes dependency on this metadata along with customer code running inside the VM</li><li><strong>P</strong>artition: The network is highly partitioned as is evident from the diagram above</li></ol><p>Metadata of virtual machines is updated less frequently, however is used heavily across the stack (reads are much more common than updates). We needed to guarantee very high availability over a very highly partitioned infrastructure. We chose to optimize on partition tolerance and availability with eventual consistency. With that having a regional service was immediately discarded because it is not possible to provide high enough availability with that model.</p><p>Coupled with the above requirements and our existing engineering investments we chose to go with approach #3 of running IMDS as a micro service on each Azure host machine.</p><ol><li>Data is fetched and cached on every machine, which means that data is lower in liveliness but is always eventually consistent as data gets pushed into those machines. Varying levels of liveliness exists based on what specific source the metadata is fetched from. Some metadata anyway needs to be pushed into the machine before it is applied and is hence always live, others like say VM tags has lower liveliness guarantee</li><li>Since the data is served from the same physical machine, the call doesn’t leave the machine at all and we can provide very high availability. Other than ongoing software deployments and system errors the data is always available. There is no network partition.</li><li>There is no need to further balance load or shard out data because the data is on the machine where it is being served. The solution automatically scales with Azure because more customers means more Azure machines running them and more placed IMDS can run on</li><li>However, deploying and telemetry at this scale is tough. <a href="https://azure.microsoft.com/en-us/regions/">Imagine</a> how large Azure deployment is and consider deploying and updating a service that runs everywhere on it. </li></ol><p>It’s really fun working on problems on this scale and it’s always a learning experience. I look forward to share more details on my blog here.</p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-31520795379966806252017-09-08T06:01:00.000-07:002017-10-16T01:56:18.442-07:00Azure Instance Metadata Service<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-CFTcx50xpNtOE9gjklOSKOhNW10PmmZUHxZYTwd3fzJC_HTKPOIB9QD97F3HDcUDmSz_lPpjLr-MtGgIAqh0TwWagsq4JA1glc-8XTrAe27wEoZ6l98YQFAUUregstkOoLw_FQ/s1600-h/image%255B6%255D"><img width="493" height="309" title="image" style="border: 0px currentcolor; border-image: none; margin-right: auto; margin-left: auto; float: none; display: block; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8nM68_fqWWGsK2R-nqlvQaAXKaf7TEvUSlDizZvU86safiIebZQsRO2_-MPbVdoaJDVwhGtmnW8mQz5GuSTE9m9m2Oq85_dvio0oBSkbttb9fYvHfqtFzz-ZrqPJ50xOacMGSlQ/?imgmax=800" border="0"></a></p><p>One of the projects in Microsoft Azure that I have been involved with is the instance metadata service (IMDS) for Azure. It’s a massively distributed service running on Azure that among other things brings metadata information to IaaS virtual machines running on azure.</p><p>IMDS is documented at <a href="https://aka.ms/azureimds">https://aka.ms/azureimds</a>. Given that the API is already well documented at that location and like all services will evolve to encompass more scenarios in the future, I would not repeat that effort here. Rather I wanted to cover the background behind some of the decisions in the API design.</p><p>First lets look at the API itself and break it down to it’s essential elements</p>
<pre>D:\>curl -H Metadata:True "http://169.254.169.254/metadata/instance?<br>api-version=2017-04-02&format=text"
compute/
network/
</pre>
<p>Metadata API is REST based and available over a GET call at the non-routable IP address of 169.254.169.254. This IP is reserved in Azure for some time now and is also used for similar reasons in AWS. All calls to this API has to have the header Metadata:True. This ensures that the caller is not blindly forwarding an external call it received but is rather deliberately accessing IMDS. </p><p>All metadata is rooted under /metadata/instance. In the future other kinds of publicly available metadata could be made available under /metadata.</p><p>The Api-versions are documented in the link shared above and the caller needs to explicitly ask for a version, e.g. 2017-04-02. Interestingly it was initially named 2017-04-01, but someone in our team thought that it’s not a great idea to ship the first version of an API based on April fools day.</p><p>We did consider supporting something like “latest”, but experience tells us that it leads to fragile code. As versions will be updated, invariably some user’s scripts/code depending on latest to be of some form breaks. Moreover, it’s hard from our side to also gauge what versions are being used in the wild as users may just use latest but have implicit dependency on some of the metadata values.</p><p>We support two formats, JSON and text. On using JSON you can fetch the entire metadata and parse it on your side. A sample from Powershell screen shot is shared below.</p><p><a href="https://lh3.googleusercontent.com/-qR8y1pvUlvs/WbKeoRAaenI/AAAAAAAAHaE/dkDcliuVy3wbpwbKX5EjY2jp6MLBcccfQCHMYCw/s1600-h/image%255B25%255D"><img width="640" height="185" title="image" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://lh3.googleusercontent.com/-Hw-yCpzKAjE/WbKeqQqrKYI/AAAAAAAAHaI/TyP1pCK3QucasPE-4NVOUA-XF-3ijwYQQCHMYCw/image_thumb%255B16%255D?imgmax=800" border="0"></a></p><p>However, we wanted to support a simple text based approach as well. It’s easiest to imagine the metadata as a DOM (document object model) or even a directory. On asking for text format at any level (the root being /metadata/instance) the immediate child data is returned. In the sample above the top level compute and network is returned. They are each in a given line and if that line ends with a slash, it indicates that the data has more children. Since compute/ was returned we can fetch it’s children by the following.</p>
<pre>D:\>curl -H Metadata:True "http://169.254.169.254/metadata/instance/<strong>compute</strong>?api-version=2017-04-02<br>&format=text"
location
name
offer
osType
platformFaultDomain
platformUpdateDomain
publisher
sku
version
vmId
vmSize
</pre>
<p>None of them have a “/” suffix and hence they are all leaf level data. E.g. we can fetch the unique id of the VM and the operating system type with the following calls</p>
<pre>D:\>curl -H Metadata:True "http://169.254.169.254/metadata/instance/<strong>compute/vmId</strong>?<br>api-version=2017-04-02&format=text"
c060492e-65e0-40a2-a7d2-b2a597c50343</pre>
<pre>D:\>curl -H Metadata:True "http://169.254.169.254/metadata/instance/<strong>compute/osType</strong>?<br>api-version=2017-04-02&format=text"
Windows</pre>
<p>The entire idea being that the API is usable from callers like bash-scripts or other cases that doesn’t want or need to pull in a JSON parser. The following bash script pulls the vmId from IMDS and displays it</p>
<pre>vmid=$(curl -H Metadata:True "http://169.254.169.254/metadata/instance/compute/vmId?<br>api-version=2017-04-02&format=text" 2>/dev/null)
echo $vmid</pre>
<p>I have shared a few samples of using IMDS at <a title="https://github.com/bonggeek/Samples/tree/master/imds" href="https://github.com/bonggeek/Samples/tree/master/imds">https://github.com/bonggeek/Samples/tree/master/imds</a></p><p>Do share feedback and requests using the URL <a href="https://aka.ms/azureimds">https://aka.ms/azureimds</a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-26533236735345931912016-11-13T16:51:00.001-08:002016-11-13T16:51:13.094-08:00Magic Mirror<p>I just implemented a MagicMirror for our home. Video footage of it working.</p> <p> <div id="scid:0ABB7CC8-30EB-4F34-8080-22DA77ED20C3:669ea267-6989-42ba-a112-fafaee2ab03c" class="wlWriterEditableSmartContent" style="float: none; padding-bottom: 0px; padding-top: 0px; padding-left: 0px; margin: 0px; display: inline; padding-right: 0px"><div><object width="566" height="318"><param name="movie" value="http://www.youtube.com/v/mCrHooiS1vM&hl=en"></param><embed src="http://www.youtube.com/v/mCrHooiS1vM&hl=en" type="application/x-shockwave-flash" width="566" height="318"></embed></object></div></div></p> <p>Basically I used the open source <a href="http://github.com/MichMich/MagicMirror">http://github.com/MichMich/MagicMirror</a>. I had to do a few changes. I added a hideall module so that everything is hidden and comes visible on detecting motion. My sources are at <a title="https://github.com/bonggeek/hideall" href="https://github.com/bonggeek/hideall">https://github.com/bonggeek/hideall</a></p> <p>This is the first prototype working on desk monitor with a 1mm two-way mirror held in front of it.</p> <p><a href="https://lh3.googleusercontent.com/-tLTxx9X5nu8/WCkKbvJqngI/AAAAAAAADB0/3Kvq-JsDix4/s1600-h/20161101_014220%25255B4%25255D.jpg"><img title="20161101_014220" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="20161101_014220" src="https://lh3.googleusercontent.com/-ASOGp1TazZo/WCkKcXeIgjI/AAAAAAAADB4/Z3LnTjYxKp4/20161101_014220_thumb%25255B1%25255D.jpg?imgmax=800" width="135" height="240"></a></p> <p>It was evident that the thin 1mm acrylic mirror is no good, because it bent easily giving weird distorted images. I moved to a 3mm 36” by 18” mirror and started working on a good sturdy frame.</p> <p><a href="https://lh3.googleusercontent.com/-ynfkMJbvzG0/WCkKciTvSpI/AAAAAAAADB8/yv3VQCCkYJs/s1600-h/20161105_154428_Richtone%252528HDR%252529%25255B4%25255D.jpg"><img title="20161105_154428_Richtone(HDR)" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="20161105_154428_Richtone(HDR)" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCwJt6L148bS3belGjDrWNCfWfI_G8IXPyzDFRI0YHxflQlcfXeWGQQlqg0Jxb_kOzcPBvrBKIFf8yPjBgBM5GlDLTvRR-rAGXmVgaArn5OEY-NOxJCyIbJS0ku_13abMxT2LErg/?imgmax=800" width="236" height="434"></a><a href="https://lh3.googleusercontent.com/-V2fvsNCbdcs/WCkKdJoLGvI/AAAAAAAADCE/LUzEGtLmHbo/s1600-h/20161105_154503_Richtone%252528HDR%252529%25255B4%25255D.jpg"><img title="20161105_154503_Richtone(HDR)" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="20161105_154503_Richtone(HDR)" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLoNRFArlMlSToYJJyoD_dlhwth-2Z-ywzywUwO3_iUcKPJDlpThfWm3EkOmIUqF_Nxikg5x9imyuqCyEaRN0is4k7dRt5iG2um7Un_is4PKFdvGIri67fgVLzlED84QSS7JN-NQ/?imgmax=800" width="237" height="435"></a></p> <p><a href="https://lh3.googleusercontent.com/-3yTSuKjxGfY/WCkKdh32jQI/AAAAAAAADCM/sl8qe7wDuBs/s1600-h/20161105_160518%25255B5%25255D.jpg"><img title="20161105_160518" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="20161105_160518" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAbc0g4F-u5PejHH7Dvie05Bgvrj0FWN3k1hbCFVKh0dIRDOQwZJru-tvAksO060iKQPjzZCF-qSH1TX9DOpkEjjFIDmPu6UzXpCZpNDv642w1jrhHc3LwyXX_l76V3hZ25qCIJg/?imgmax=800" width="470" height="280"></a></p> <p>I used 3x1 wood for the size which is <a href="http://diyhousetips.com/314/nominal-lumber-size-dimensions-chart/">actually</a> 2.5 x 0.75 inches. On that I placed a face-frame.</p> <p>I had a smaller 27” old monitor and decided to just use that. I mounted and braced the monitor with L shaped brackets. So it is easy to take out as well as hold the monitor firmly in place.</p> <p><a href="https://lh3.googleusercontent.com/-2bQPGqOi2U0/WCkKeVnlYSI/AAAAAAAADCU/LIp89C2gXFg/s1600-h/20161107_163647%25255B5%25255D.jpg"><img title="20161107_163647" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="20161107_163647" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi7T6gd_C4VvafrSLgQ4Orrf3MdpZEZhCNwx7BSwVsxc9boudRbc3beUc8Nz45cKRuIeMFN_XvXTEY2UGRqS6JBdpvgSx7lHoC4AMCbVjj-CCjz8CuXgpZK6nocLk_kibfGj4Rmw/?imgmax=800" width="238" height="437"></a></p> <p>Final pictures</p> <p><a href="https://lh3.googleusercontent.com/-mZvY2x0aKpA/WCkKezZeMZI/AAAAAAAADCc/hu1dMRjq3XM/s1600-h/IMG_8263%25255B5%25255D.jpg"><img title="IMG_8263" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="IMG_8263" src="https://lh3.googleusercontent.com/-7BKYv0FXt1M/WCkKfFO3z5I/AAAAAAAADCg/psJiw_t0NwE/IMG_8263_thumb%25255B2%25255D.jpg?imgmax=800" width="186" height="288"></a><a href="https://lh3.googleusercontent.com/-84igaSqZNuw/WCkKffFgXRI/AAAAAAAADCk/aaXhskeB_pI/s1600-h/IMG_8252%25255B5%25255D.jpg"><img title="IMG_8252" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="IMG_8252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjloOvGuyZN3BdJFUxRk78J70s0b_UD4jD-HMjmrpHL7CBV_y2Eqd_SFz6nGQmLXNhcf6VMrhkadeGKwBCUVQSimuouxU4j_Lo1Y5B0_tKyhVckT4mu51ppEF-a7Kz-7p-eKHcUEw/?imgmax=800" width="186" height="288"></a><a href="https://lh3.googleusercontent.com/-ATVL09Dn6DM/WCkKfxN91_I/AAAAAAAADCs/8xOMgvAnQIs/s1600-h/IMG_8253%25255B5%25255D.jpg"><img title="IMG_8253" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="IMG_8253" src="https://lh3.googleusercontent.com/-oZ4UjgRUwcE/WCkKgKtIQcI/AAAAAAAADCw/C8inDYNqaNs/IMG_8253_thumb%25255B2%25255D.jpg?imgmax=800" width="186" height="288"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-23957303389748774512016-06-22T14:30:00.000-07:002016-06-22T14:55:39.834-07:00Caribbean Vacation planning in Summer<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia-_IG0gQ69XaFXsj_UrI-vAvD_0wStw0td4pnntYQsI8ICLXPs2zVmkx_9mySyaRNMd0QjivOo_XeHHoHa92TPqleLokMxjhwPYlnycNIOh7n_EkgSGbOs7TU-_YQt_wwSlY0CQ/s1600-h/13497812_10153551994327274_9173203905237221898_o%25255B6%25255D.jpg"><img title="13497812_10153551994327274_9173203905237221898_o" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="13497812_10153551994327274_9173203905237221898_o" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiipdOfuBqM5-kw7rL4m_TXWw6ZG99dFz7fTLyYxiiJ_zbwE32ucsQcgoGSuEtSvqUGT6rqmvevXnbwAD2IwHOq1ZX3wvWLqIp8GSDUYp4sTisBLGdgvXR26XodWkNfs5uCeLh0RQ/?imgmax=800" width="585" height="389"></a></p> <p>I just came back from a vacation in Aruba. A friend asked me why Aruba as he thought is was an unusual choice.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj21sjfBmOZkYuIMDKKRkUqUjk71y3pjZV8UxMCTOvW_ktBntdZjvx0Y0Nm-JHyxxwS6RQXkPPi_Yt57WfOLibxziCAfmW1qZaNHZEpk40tkrG7IXhZzt4IVK2OgEZqlIUZIqalZg/s1600-h/image%25255B31%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; float: none; padding-top: 0px; padding-left: 0px; margin-left: auto; border-left: 0px; display: block; padding-right: 0px; margin-right: auto" border="0" alt="image" src="https://lh3.googleusercontent.com/-QDZFXUgBiJA/V2sJWwv0mZI/AAAAAAAAC88/Y0Li12VW1Ws/image_thumb%25255B19%25255D.png?imgmax=800" width="327" height="252"></a></p> <p>I was trying to explain how my choice had a large data driven approach. Then I thought I’d share how I do it.</p> <h2>Why Now</h2> <p>First of lets get the constraints out of the way. Because choosing between “where to go at this time” vs “when to go to place X” needs different data crunching. I had to go now, the time is not a choice and is governed by the fact that I had manageable work load, it is summer and school vacations are on(I cannot go when it is not), I cannot go later during summer because of other engagements. Also I wanted to go relax in a Caribbean destination. So the question came down to where in the Caribbean and I used historical data to make the choice. And boy did that work out!! </p> <h2>Weather Data</h2> <p>First of it is hurricane season in the Caribbean so my first quest was to find a place where I am least likely to be hit by one. For that I headed onto the NOAA site <a title="https://coast.noaa.gov/hurricanes/" href="https://coast.noaa.gov/hurricanes/">https://coast.noaa.gov/hurricanes/</a> which maps all hurricanes at a given place for the last 100 or more years. Choosing Bahamas and then all hurricanes only (not even storms) for June/July shows me the following. So you can make it out that it is a super bad idea.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhksD5j-xh0B__XW1iytWB0bd1mYdV6Zo4CL0eVOHY3pq3c9_x7rSSLhZoMy_lNQptQGzULj6GQeX37WF5yq6zcl_EslwIpBSrgzij84BkNrPVTCb3UhOUvAuXsABJ_o6UcSicNfw/s1600-h/image%25255B15%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-sE_VjS_Lboz5J6VMTmKbusDuKQLp8Wp9Z06GPXDCGvWL7UM06604JbwzWuw2wrYJA4SbPuz9i-XBxpJQlluVueY0dd51Q7owOGTJIWKhxKz-uuovK3oOGgiToWOwoG1yskKmqg/?imgmax=800" width="533" height="435"></a></p> <p>I did the same search for other popular destinations in that area. E.g. Aruba for the last 100 years, shows not a single one hit this island.</p> <p><a href="https://lh3.googleusercontent.com/-_c0P9VoYUbg/V2sDgZDmGjI/AAAAAAAAC78/u-1GOFNBHEQ/s1600-h/image%25255B26%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="image" src="https://lh3.googleusercontent.com/-DOzCwNMTOmE/V2sDgxxSsMI/AAAAAAAAC8E/2k8SMWJmlEY/image_thumb%25255B16%25255D.png?imgmax=800" width="580" height="461"></a></p> <p>Now that we know a hurricane doesn’t generally hit this place, lets see precipitation data.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx6uaE61fomUbai2ppV2cGZn9RLiehpEU_1H2hO_1TwZn3dySFxFPS3SbEHshhb45aEI__fh2NaV-PWvIDakRmDrjUGH4-beRm6SQuEG12YPpyx0y_7JVyddCZr0nhcYZh7j9ULA/s1600-h/image%25255B25%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="image" src="https://lh3.googleusercontent.com/-Nf6ZzxcukZY/V2sDha84A0I/AAAAAAAAC8U/BLOjvkCCAvQ/image_thumb%25255B15%25255D.png?imgmax=800" width="517" height="335"></a></p> <p>So it’s very unlikely to rain either. Wow I am sold already.</p> <h2>Other Data</h2> <p>Obviously weather is not the only criteria. I did similar search with crime rates, prevalence of diseases etc. And then when I narrowed down to the area around the ABC islands (Aruba, Curacao, Bonaire), I finally made my choice on Aruba based on the kind of prices I got on hotels.</p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-6528341487865164702016-05-04T11:25:00.001-07:002016-05-04T11:25:27.148-07:00Customizing Windows Command Shell For Git session<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif0RW8BQXc5XFJuXIjzt47xFFULKRHlcM_vHqbEzPARwufm2VnpywtmwYYdkDkYWmJyKbh1MVfgwgS96SvK3fNYxkALOP5p3lIbCY0CSgQ5ik5kax6HsYaoa-pDaYPulzCZN09iw/s1600-h/23713373012_50e0f75403_z%25255B4%25255D.jpg"><img title="23713373012_50e0f75403_z" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="23713373012_50e0f75403_z" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBDja4ZIFW7Tq6jTj7FMlRC1YTBSLSeQ64YN3nU8AfO9bpqOdWFeIP0d1-lhgr1VldZQX86HYH3IMFnXeKznoUlECAfZ0BYxAJf0zlq_ItRm2A8yxJrubqi6YRLJSW2YSq5ICqNA/?imgmax=800" width="560" height="392"></a></p> <p>Old habits die hard. From my very first days as developer in .NET and Visual Studio , I am used to have my windows cmd shell title always show the branch I am working on and have a different color for each branch. That way when I have 15 different command window open, I always know which is open where. Unfortunately when I recently moved to GIT, I forgot to customize that and made the mistake of making changes and checking in code into the wrong branch. So I whipped up a simple batch script to fix that.</p><pre>@ECHO OFF
git checkout %1
for /f "delims=" %%i in ('git rev-parse --abbrev-ref HEAD') do set BRANCH=%%i
@ECHO.
title %BRANCH%
REM Aqua for branch Foo
if "%BRANCH%" == "Foo" color 3F
REM Red for branch bar
if "%BRANCH%" == "Bar" color 4F
REM Blue
if "%BRANCH%" == "dev" color 1F</pre>
<p>I saved the above as co.bat (short for checkout) and now I switch branches using the co <branch-name>.</p>
<p>You can see all the color options by running color /? in your command window. </p>
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKp3VjVJHbs28YyBGCMRT1SPqsi5cJKNRAiS7FtREqrCZwna-cRtkyiJ71y7U6IBaZLEN782FntneebW_f_gfp4LFlFXQbC9yNtYooQZEB2vOScsY9FURhFk8PfxXc3nDunsvD4g/s1600-h/image%25255B4%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="image" src="https://lh3.googleusercontent.com/-0zk4pKr7YD0/Vyo-lbTA3VI/AAAAAAAABQ0/wDmUQFiSRBs/image_thumb%25255B2%25255D.png?imgmax=800" width="579" height="302"></a></p>
<p>or</p>
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLYSFMn6dfI87rHa0GuWAMJIEf5470WOWag3Qskuv7JbM03Fe5t9I57dGSqz1Qs7rVx3DhKzFIk6k9TkD13Vc-xpahULYXMhJIBqwlvZhNviPhBYKdtAHS9yrkEpdlnosQrm1aCA/s1600-h/image%25255B10%25255D.png"><img title="image" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPJncWzM5zi2D16mGLmZTni415sez6Jc6aZI42is8f6toop5p41mv-0F4x2iL8goTd6KPPaVCFc3PX_je0Hs2m9e_A_ygwgGYveHk-hQmltg-zo0NSKQ2GVygHOSrC3iW2tG8Fig/?imgmax=800" width="583" height="321"></a></p>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0tag:blogger.com,1999:blog-6434675.post-38213224407224758922016-02-21T16:18:00.001-08:002016-02-21T16:18:36.709-08:00Identifying your Arduino board from code<p><a href="https://www.flickr.com/photos/abhinaba/23238553810/"><img src="https://c2.staticflickr.com/6/5684/23238553810_389fe4e177_z.jpg" width="551" height="443"></a></p> <p>For my IoT project I needed to write code slightly differently for specific Arduino boards. E.g. for Arduino UNO I wanted to use Serial to talk to ESP8266 and for Arduino Mega wanted to use Serial1. So basically I wanted to use Board specific #defines </p><pre>#ifdef MEGA
#define SERIAL Serial1
#elif UNO
#define SERIAL Serial
#endif
SERIAL.println("Something")
</pre>
<p>For that I needed to get board specific #defines for each of the board options in the Arduino IDE, so that as I change the board type in the IDE, I automatically build code for that specific board.</p>
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrIkz3H2C-cHMmFBtpVEo4OpHhuWAUpuvOGAwTTbGZ46YR2u-MA05F0XDHEUqJNAmEPqI17q_xF__msLIpzhgH_E5EaqU5d5hA90xtC94mhzdl8RjnU7NTatXk_0Jen2ykpmr2Vg/s1600-h/image%25255B5%25255D.png"><img title="image" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqGiwV2F_3GLazE-6wJEj33_0zDX6sWMmqVy5a8AC88RCFL5hDqeZIUzeMcGgZUNkc8sb5gbuj0CpP8fwH-YsFSDND5QsnHHyUdMS6jhtT9PwnCDXPV-UHKUcSOWDSnbSTBL7YaQ/?imgmax=800" width="585" height="505"></a></p>
<p>That information is actually available in a file called board.txt inside your arduino install folder. For me it is G:\arduino-1.6.5\arduino-1.6.5-r5\hardware\arduino\avr\boards.txt. For each board there is a section inside that file and the relevant entries look something as follows</p><pre>##############################################################
<strong>uno.name=Arduino/Genuino Uno</strong>
uno.vid.0=0x2341
uno.pid.0=0x0043
uno.vid.1=0x2341
uno.pid.1=0x0001
...
<strong>uno.build.board=AVR_UNO</strong>
uno.build.core=arduino
uno.build.variant=standard
</pre>
<p>The .board entry when prefixed by ARDUINO_ becomes the #define. I wrote a quick PowerShell routine to get all such entires. The code for it is in GitHub at <a title="https://github.com/bonggeek/Samples/blob/master/Arduino/ListBoard.ps1" href="https://github.com/bonggeek/Samples/blob/master/Arduino/ListBoard.ps1">https://github.com/bonggeek/Samples/blob/master/Arduino/ListBoard.ps1</a></p><pre>$f = Get-ChildItem -Path $args[0] -Filter "boards.txt" -Recurse
foreach($file in $f)
{
Write-Host "For file" $file.FullName
foreach ($l in get-content $file.FullName) {
if($l.Contains(".name")) {
$b = $l.Split('=')[1];
}
if($l.Contains(".board")) {
$s = [string]::Format("{0,-40}ARDUINO_{1}", $b, ($l.Split('=')[1]));
Write-Host $s
}
}
}
</pre>
<p>Given the argument of root folder or Arduino install, you get the following.</p><pre>PS C:\Users\abhinaba> D:\SkyDrive\bin\ListBoard.ps1 G:\arduino-1.6.5\
For file G:\arduino-1.6.5\arduino-1.6.5-r5\hardware\arduino\avr\boards.txt
Arduino Yún ARDUINO_AVR_YUN
Arduino/Genuino Uno <strong>ARDUINO_AVR_UNO</strong>
Arduino Duemilanove or Diecimila ARDUINO_AVR_DUEMILANOVE
Arduino Nano ARDUINO_AVR_NANO
Arduino/Genuino Mega or Mega 2560 <strong>ARDUINO_AVR_MEGA2560</strong>
Arduino Mega ADK ARDUINO_AVR_ADK
Arduino Leonardo ARDUINO_AVR_LEONARDO
Arduino/Genuino Micro ARDUINO_AVR_MICRO
Arduino Esplora ARDUINO_AVR_ESPLORA
Arduino Mini ARDUINO_AVR_MINI
Arduino Ethernet ARDUINO_AVR_ETHERNET
Arduino Fio ARDUINO_AVR_FIO
Arduino BT ARDUINO_AVR_BT
LilyPad Arduino USB ARDUINO_AVR_LILYPAD_USB
LilyPad Arduino ARDUINO_AVR_LILYPAD
Arduino Pro or Pro Mini ARDUINO_AVR_PRO
Arduino NG or older ARDUINO_AVR_NG
Arduino Robot Control ARDUINO_AVR_ROBOT_CONTROL
Arduino Robot Motor ARDUINO_AVR_ROBOT_MOTOR
Arduino Gemma ARDUINO_AVR_GEMMA
</pre>
<p>So now I can use</p><pre>#ifdef ARDUINO_AVR_MEGA2560
// Serial 1: 19 (RX) 18 (TX);
#define SERIAL Serial1
#elif ARDUINO_AVR_NANO
#define SERIAL Serial
#endif // ARDUINO_MEGA</pre>Abhinaba Basuhttp://www.blogger.com/profile/17653254493243741669noreply@blogger.com0