Since I was a kid, I have made figurines from found objects: shells, stones, twigs, wire, and wood. A few years ago, while wrapped in a sleeping bag in Geronimo Cave in the Superstition Wilderness near Phoenix, Arizona, I made “Avi” for a dear friend.
This evening I hosted a figurine party. We assembled just before sunset to gather shells, stones, and sand from the Muizenberg beach. A few pizzas, bottles of wine, and tubes of superglue.
This is what transpired.
So goofy I just had to share this “Machine Learning A Cappella – Overfitting Thriller!”
On Wednesday, June 24 I will address the Astronomical Society of South Africa, at the SAAO auditorium.
The official write-up was presented as follows:
Kai Staats will give a presentation entitled “From Dark Skies to Data Mining: A Propensity for Pattern Recognition”. The talk will address the situation where few people today are able to experience the brilliance of the milky way due to light pollution and yet our window on the universe is expanding enormously through projects such as the SKA and LIGO.
We are faced with the challenges of fighting to preserve dark skies and at the same time enjoying the benefits of the massive quantities of data becoming available. He will then screen his film “LIGO, A Passion for Understanding”. The Laser Interferometer Gravitational-Wave Observatory, a large-scale physics experiment, is aimed at directly detecting gravitational waves. These ripples in the space-time, known as ‘The Big Bang’s Smoking Gun’ were predicted by Einstein in 1916 and will provide detailed information
about black holes as well as the very early universe. Marco Cavaglia, astrophysicist and member of the LIGO Collaboration will be available, via Skype, to answer questions following the film.Kai Staats is an entrepreneur, writer, film maker and now student once more, earning his MSc in Applied Mathematics at the University of Cape Town / African Institute for Mathematical Sciences, Cosmology Research Group, South Africa.
Here in Madrid, Spain, I reunited with Dalia, a friend I have not seen since 2004 in the little mountain town of Vinales, Cuba.
Between the two of us, we recalled so many splendid moments in those distant days, when the call of the limestone climbing was all that mattered. I would sharpen a machete each morning and march through farmers’ fields, cutting pathways to caves and boulders. The Cubanos on their way to work, standing in the back of a Russian dump truck or crammed too tight in an old bus, would pass and yell to me, wondering why this American was carrying a machete and where he could be going.
Each weekend my Cuban friends would follow me to those hollow domes, to be the first to climb the new problems I had set. Do you remember “La vangenza de Ana Maria”? A mossy, slippery, nearly green wall at the back of the cave. Ana Maria, the smallest of us all, was the first to make it to the top without falling.
When the sun had set, we filled the plastic containers my host family daily provided for my lunch with rocks and sand, marching down the middle of the road by moonlight. We danced and sang to the rhythm of shakers and shoes against pavement.
Once we were camped at a home not registered as a casa particular and a neighbour called the police, a typical event in a totalitarian state. Anibal’s aunt would be fined, jailed, or worse–if caught with Americans in her home. Taralee and I jumped in the back of a dump truck and sped off to a designated intersection marked by a large tree, an hour down the road. We were reunited later that day and Anibal’s aunt was unscathed.
Posterboy Anibal, Russo, Renier, Dalia and Elaine, El Turbo and Anna Maria, the German Tom and Canadian Devin. What an eclectic crew, like none before and none to come again. Dalia confirmed this evening, as we walked to catch her bus, “You came at a special time Kai. Those days were very unique. Nearly everyone we knew left Cuba and Vinales was never the same.”
Those moments came to life for me again, like something out of a story book.
Thank you Dalia, for conversation and story time.
(email to my fellow researchers)
This morning I completed the final evolutionary process for my GP code. It now supports Reproduction (no mutation), Point Mutation, Branch Mutation, and Crossover Reproduction.
I have run thousands of trees through the system without a glitch. In my new “debug” mode I am able to visually monitor the expansion and pruning of GP trees to make certain they are evolving as required.
I have learned the Sympy library which takes my full strings and reworks them to functional, more compact functions is not terribly forgiving. Yes, it will take a few dozen elements in a long polynomial and reduce the complexity (removing two instances of a variable which cancel each other, or rewording multiple instances as 2x, 3x, etc.), but if there is an error it simply stops execution.
This works to my favour, as I am able to trust the expressions are solid and fully functional in each generation.
I am a bit OCD so I will surely spend time making certain the internal documentation is clean and well stated. This will become the foundation for my User Guide and appendix in my Thesis.
The next major step is to run a few benchmark problems through my code to make certain I am seeing the desired output. Then, move into working again with the KAT7 data.
I am excited to get back to literature review, my thesis, and not programming every day :)
Thanks for all the support this past 3 months!
kai
If you have not watched Ex Machina — it is a must see. Sci-fi fan or not, it is a perfectly produced film. Simple plot. Excellent camera work. Incredible acting.
It gives you a sense of the very real issues we must tackle … if … when we finally give rise to the machines (which remains on a distant horizon).
I have been coding for nearly three months, with but a ten days break. I love this time and space. Singularly focused. Only one task in life. I leave my phone at home, launch my development Virtual Machine (no email, no Facebook), and dive in.
My machine learning (a subset of Artificial Intelligence) code “came to life” about ten days ago when it first converged on a solution. That was an incredible feeling. Almost there! Just one more type of mutation to complete. All tools are built but this one. Then it should just work.
While I set out to build a script to work with SKA KAT7 data, my tendency toward building things BIG kicked in and my GP instead became a platform. As a result of my approach, I inadvertently developed a pointer system for managing Numpy arrays. Each column in an axis-1 array is treated as a unique object with parent/child relations to others in the array.
If I had known how hard this would be, I would have looked for alternatives or considered a different plan. But sometimes it is best to just go into a battle blind, look back, and realize how far you have come.
When I arrived to Sutherland, with eight days and nights stretched out before me with just one principal task on my agenda, I was certain the Grow method (one of two branch mutations) was in my grasp. Yes, it evaded me for that week and another after I returned to Muizenberg. In fact, it became the single most challenging part of building Karoo GP (as noted in the email to my professor and fellow researchers May 26).
Despite the myriad sketches I produced, I was unable to see the path to fruition. I must have pursued a half dozen avenues, writing hundreds of lines of code that in the end, all failed.
I began to notice a pattern in my behaviour. I would repeatedly envision a solution. From that vantage point, I could see the challenge, the path, and the conclusion. I sketched the means, to assure it was real, and then dove into the code.
Several hours, even a day or two later, I hit a wall. Not just a small barrier, nor new challenge, but an insurmountable, unavoidable wall. With each wall, I was again reminded there was no work-around, no short-cut. The only solution invoked many stages and a great deal of coding.
How could I not have seen this? Why had this only now become no-go when just minute before, it was still possible?
I drew more sketches, more flow-diagrams, and arrived to more solutions. I even came back to an already failed solution. Having forgotten why it failed, I felt sure I must have missed something and dove back in only to be met with the same, obvious wall.
I felt stupid. Frustrated beyond belief. What’s more, my clarity of vision was diminishing rapidly. I could no longer see a half dozen steps down the coding path, but only two or three. My reduced exercise, pulling late nights with the astronomers, and change in diet all contributed.
I recall a high school art class in which we followed a book called “Drawing on the Right Side of the Brain by Betty Edwards” by Betty Edwards. One chapter advised us to hold a pencil or pen in our favoured hand, brainstorm and produce thumbnail sketches until our ideas ran dry. We felt we had surely exhausted all the possibilities for that concept.
Then we grasped the pen with the other hand, even if awkward to the majority of us who are not ambidextrous. More ideas came! Ideas from a completely different foundation.
I knew enough about my internal processes, having spent much of my undergraduate program in Industrial Design in problem solving, to recognise that I would benefit from working on something else, something that came to me more easily. I improved the interface, cleaned up the Python methods, and developed small, useful tools such as a method for tree population renumbering, another for renumbering columns in an axis-1 Numpy array, each of which represented a node in a GP tree; and ultimately, a method for tracking the differences between original and skewed node positions as a branch was inserted into a tree.
When I returned to Muizenberg and my office with a blackboard, I drew my diagrams much larger, using my entire upper body to bring form to my visions. I was able to see the solutions more clearly than when at Sutherland.
Yet, repeatedly, when I turned to sit at my computer, the blackboard to my back, the clarity vanished quickly. I’d start typing and … hit a roadblock, or completely forget what was to happen next.
I thought I was losing my mind. I could see two, sometimes three steps at a time, but four or more and I failed miserably. Endless loops of indigestible code, a headache, backache, and neck filled with anxiety.
I returned to the blackboard, erased everything, and started over. I took the sketches deeper, down to the pseudo-code level such that I was writing loose versions of Python directly on the board.
Those tiny tools I had developed in Sutherland became invaluable, for in each was a means by which I could relieve my mind of that function, trusting the power of a tested black box. The more tools I developed, the more I could focus on flow instead of the detailed interaction of ever level of code.
Success came to me when I broke a complex problem into the smallest pieces possible, connected those pieces with each other, and then validated the flow of data across the system. From this came the “debug” mode, not integral to Karoo GP as a means by which the user can monitor the mutations in real-time.
It’s fascinating to watch, even to me.
(email to my fellow researchers)
A week has come and gone. And with it, some 60 hours coding, if not more. I have accomplished a great deal, further fine-tuning the code, building tool-sets, and building a stronger method-based platform for GP. I can now build a Tree from the command line in 5 lines of code, evaluate for fitness, mutate, and save with a few more.
While I witnessed my GP converge on a solution a few nights ago for the first time, using only reproduction (copy) and simple point mutation, I have not yet completed crossover nor the grow method for branch mutation.
These have proved to be incredibly, mentally challenging. I have never struggled with something at this level (I want to say “in my life,” but I may not be thinking clearly :)
I freely admit the issue is not one of Python, but an incredibly complex array book keeping effort that borders on my having to create my own pointer system.
I spent 3 days exploring work-arounds, tricks, and short-cuts, but in the end, am back to the realisation there is just one way–the hard way.
The good news is that I have slowly eroded the challenge by building a suite of tools, each of which performs a single task. Cool.
I will get there, one step at a time.
Today was lost to the realisation (and subsequent rewrite) that my core Tournament code was (inadvertently) taking advantage of Python’s ability to perform basic comparisons (>, <, =) on strings which contains numbers, as compared to converting those to integers first. It works, but the results are not consistent. Fixed! I am far more confident, even though another day behind my slated sched.
What’s more, Python’s numpy random generator is clearly NOT random by any stretch of the imagination. 100 Trees evolved over 10 generations in non-interactive mode generates far, far more positive results than when I enter interactive mode (code pauses / continues with RETURN). This causes me to believe the random generator is using keyboard input as a random seed, which drastically (and negatively) affects the results.
I will, at another time, replace Numpy with a call to a proper, external random number generator. I see this current effort as computer science more than applied mathematics. However, what I am learning about the underlying structure of code is likely quite valuable.
Been coding till 2 or 3 am every night, photographing when the sky is clear ’till 5 or 6 am. Sleep till 10:30, go for a run, and back to the grind :)
kai
