Home - Kite’s Corner

About

Hi, I’m Tom - I explore the universe, one piece of code at a time.

Regularisation: the strange secret behind deep learning

I am probably not alone in finding regularisation to be an almost bizarre concept when first introduced. Why would you artificially penalise your models in ways that don’t benefit the learning task? Adding an L2 term guarantees the final model will walk uphill from its otherwise optimal solution, which seems contrary to the main goal. Subjecting your model to dropout artificially strips elements of its calculation away while it works, forcing it to recover from missing information. These seemingly strange ideas are not just useful hacks that happen to work, but are actually essential to the field of deep learning. Let’s explore why and how in this post.

Exploring the Mathematics of Statistical Learning Theory

It is impossible to deny the power of neural networks – recent progress in deep learning has consolidated them as the approach of choice for many learning tasks. There is, however, still a delicate art to their design, application and training. This post will dig into statistical learning theory, which provides some theoretical backbone to the fundamental tradeoffs present in any learning task, which in turn illuminates some of the nuances present in the current machine learning landscape.

Static vs. Dynamic Polymorphism

Polymorphism is a key concept within C++. It is essential in bringing some flexibility into a safe, strongly typed language. By guaranteeing an object will respond to a certain interface, you can build code which is ambivalent to the exact implementation details of such an object. It is well known however that flexible dynamic behavior necessarily incurs a computational overhead, forcing a compromise on speed and latency. There is however the option of static polymorphism, which can give a level of reusability and flexibility to your code without the runtime performance penalty. Instead this penalty is paid at compile time, which may be a good tradeoff in many applications.

TelegramGPT

Okay, I guess GPT was already in your pocket - but not as easy as this! After getting tired of constantly reminding openAI that I am indeed a human, I decided to take matters into my own hands. Hosting a personal Telegram bot on a Raspberry Pi which calls the API of the world’s new favorite AI friend seemed like a worthwhile project. Also, it is a good intersection of various buzzwords, and is therefore also a worthwhile blog post.

cpp_grad: a first adventure into backpropagation and neural networks

When I discovered Andrej Karpathy’s tutorial for micrograd I was immediately hooked. In my experience, it is rare to find resources on neural networks that build the code from scratch. Andrej Karpathy, however, does a great job of explaining the fundamentals and using simple yet powerful Python code to demonstrate the principles of backpropagation. I wanted to implement the code for myself and decided C++ would deliver a healthy challenge while potentially giving enough speed to make the code useful. I called it cpp_grad.