In a moment of needing a break from working on the User Guide I played with Karoo GP for 30 minutes and came back to testing cos and sin functions. They broke the Absolute Value function where Classification and Matching yet worked.

A day into the battle, a deeper issue has unfolded which keeps Karoo GP from working with any floats (although this was tested a long time ago, but with a very limited and controlled number of decimal places).

Now, I have learned even when [result = solution], it fails to match.

algo_sym a + b/c
result 0.44404973357
solution 0.4440497336

algo_sym a + b/c
result 0.690166666667
solution 0.6901666667

algo_sym a + b/c
result 2.70666517168
solution 2.7066651717

ARGH!!! Rounding errors!!!

I need to introduce careful control of the number of decimal places invoked for both ‘result’ and ‘solution’. For as much headache as this has caused, I am pleased I took that break and played around with my code again, else I would have run into this when I came back to KAT7 data.

What’s more, this might help Karoo GP solve the Kepler planet problem (which still fails at, miserably).

(sitting in the bookstore in Kalk Bay)

This is my first time working on Karoo GP since my daughter Lindah’s arrival to South Africa nearly two weeks ago. She departed just yesterday. Out time together was incredible. We both learned so much. I am so sad to see her go :(

My intent is to complete the User Guide by the close of the weekend. I was able to complete all in-line documentation for the main script. I am now in the process of completing the in-line documentation for the base_class.

In so doing, I derived the following simple but quite useful equation to determine the maximum number of nodes (assuming a Full tree) in any given GP tree:

nodes = 2^(depth + 1) – 1

For example:

Depth 1 = 3 nodes (1 functions, 2 terminals)
Depth 2 = 7 nodes (3 functions, 4 terminals)
Depth 3 = 15 nodes (7 functions, 8 terminals)
…
Depth 10 = 2047 nodes (1023 functions, 2024 terminals) *

* that is one huge-ass (scientific lingo) polynomial!

August 15-20

I was able to introduce multi-core functionality using ‘pprocess multicore library‘ (Thanks for the pointer Arun!). This effort required roughly four days research and coding, as it was my very first go. In the end, it is relatively simple, depending upon the code in which it is introduced. I included a user-defined core quantity, and the ability to modify the number of cores engaged during runtime.

The key to multi-core functionality is getting your head wrapped around protected memory spaces. That is, each core assigned by pprocess (or any multi-core library) will reproduce a fully functional copy (instance) of the section of your code which you are spawning on each core.

Each of these instances can read from any global variable, no problem. However, they cannot write back to global variables as all of them would try to write back to the same variable at the same time, known as a race condition. Bad things would happen.

It is imperative to keep in mind that what happens in each instance will not affect what happens in the other instances, on the other cores. Once spawned, they are all independent even though they work with the same variable names and are processed by copies of the same code.

If the returned values are to affect something, that something must be outside of the multi-core environment, post-collection. This may require some rearranging of your code.

For me, it meant that instead of a fitness = fitness + 1 inside the pprocess pipeline, I returned the single value of fitness for each instance on each core, and then conducted the sum function when I collect the results from pprocess.

Below I offer an example of the original for loop and the pprocess which replaces it, for multi-core support. I have left the single core for loop in place as a user invoked bypass of pprocess when the overhead of multi-core processing reduces performance (common on non-CPU intensive runs or on a limited number of cores).

In my code, each GP tree must evaluate all rows in the given data. I therefore employ a for loop to iterate through each row in previously loaded .csv file (not shown in this example). The Python method (function) fitness_eval() is what conducts the evaluations.

As fitness_eval itself calls another two Python methods (3 methods in total), the key was to rewrite my code such that any global variables for which values are updated (written to) were made local instead, passed from method to method directly. In the end, this makes for better code, forcing me to rethink the way I had designed this and others sections.

if cores == 1: # employ only one CPU core and bypass ‘pprocess’
   for row in range(0, data_rows): # increment through all rows in the data
      fitness = fitness + fitness_eval(row) # evaluate tree fitness

else: # employ multiple CPU cores using ‘pprocess’
   results = pp.Map(limit = self.cores)
   parallel_function = results.manage(pp.MakeParallel(fitness_eval))
   for row in range(0, data_rows): # increment through all rows in data
      parallel_function(row) # evaluate tree fitness

   fitness = sum(results[:]) # ‘pprocess’ returns the fitness scores in a single dump

In this example, the pprocess method parallel_function() is a wrapper for my original method fitness_eval(), such that I pass the incrementing variable row through parallel_function(row) which in turn hands if off to multiple copies of fitness_eval(). Cool, huh?! :)

One benefit of pprocess, as compared to the Python mulitiprocessing in Python library, is that pprocess can pass more than one variable through the called methods without using a pickle (a package used to send variables to and from methods while in mulit-core memory spaces). So, you can treat your methods just as you would on a single core, sending and returning variables using method(var_1, var_2, var_n) and subsequent return var_1, var_2, var_n.

But keep in mind what I shared above, about how those variables can only return an isolated value per instance, as changes to each of those variables on each core will not affect the other instances. Make sense?

Many thanks to the Stackoverflow community and Paul Boddie, co-author of pprocess.

(continued from GP update 20150813)

I see 4 ways to deal with the premature convergence:

a) Take a pill.

b) If any given tree falls below the user-defined number of nodes (node count, not depth count), that tree is forced to mutate over and over again ’till it is at or above the prescribed node count. This feels convoluted, as this is not how it happens in the biological world.

c) Nudge the fitness function (higher or lower, depending upon max or min function) such that a tree whose node count is below the user-defined number is less likely to be selected in a Tournament.

d) Simply block any tree whose node count is lower than the user-defined number from entering a tournament. As all four of my mutation types are channelled through tournament selection, this is an easy, 2 line solution.

Is it real-world? Is it any different than applying a maximum depth?

Hmmmm …

(resolved in Kepler’s Law resolved by GP!)

(email to my fellow researchers)

Subject: premature convergence

No, this email is not about some “teenage” problem :)

Per my conversation with Emmanuel today, the classic Kepler’s 3rd Law of Planetary Motion (called the “Harmonies Law”) is the square of the period divided by the cube of the mean radius from the center of the Sun to the center of the planet.

My new “minimisation’ function is working beautifully! However, maybe it works a little too well. Karoo GP is quickly converging on p/p (period divided by period) which of course equals 1, where 6 of the 9 planets are defined as 1.0 in the dataset I am using (the other 3 planets are 0.99 or 0.98).

So, I am going to introduce a new user defined “Min Nodes” which will set the minimum number of nodes (elements) in the GP tree (equation). I feel like this is cheating, but Emmanuel confirmed that most GP problems require some tweaking of the code.

There are 2 ways a large tree can very quickly get smaller: Grow Mutation or Cross-Over mutation (Reproduction, Point Mutation, and Full Mutation do not alter Tree size). I can intercept either of these and force them to evolve again and again until the new tree is above the min boundary; or simply invoke an artificial fitness boost in the right direction.

What would Darwin say? What would Gandhi do?

It seems to invoke evolutionary pressure is the one more like the “real-world”, no? To force something to evolve again and again until it satisfies a certain criteria is a bit convoluted. But any more so than defining a maximum depth?

We’ll see how I feel in the morning :)

kai

(continued in Premature convergence)

(sitting at SKA)

Struggling to get back into my code.

Preparing to test Kepler’s 3rd Law of Planetary Motion. This is a minimisation problem, meaning the best overall fitness will be the smallest number. I will employ the Absolute Value Difference fitness function I recently developed. Simple in theory, but always a few hidden challenges in implementation.

(later)

Realised I need to re-think all my fitness functions, and embed both minimisation and maximisation into the tournament and ‘fitness gym’. Will move to define each fitness kernel as ‘min’ or ‘max’ at the opening.

So goofy I just had to share this “Machine Learning A Cappella – Overfitting Thriller!”

(email to my fellow researchers)

This morning I completed the final evolutionary process for my GP code. It now supports Reproduction (no mutation), Point Mutation, Branch Mutation, and Crossover Reproduction.

I have run thousands of trees through the system without a glitch. In my new “debug” mode I am able to visually monitor the expansion and pruning of GP trees to make certain they are evolving as required.

I have learned the Sympy library which takes my full strings and reworks them to functional, more compact functions is not terribly forgiving. Yes, it will take a few dozen elements in a long polynomial and reduce the complexity (removing two instances of a variable which cancel each other, or rewording multiple instances as 2x, 3x, etc.), but if there is an error it simply stops execution.

This works to my favour, as I am able to trust the expressions are solid and fully functional in each generation.

I am a bit OCD so I will surely spend time making certain the internal documentation is clean and well stated. This will become the foundation for my User Guide and appendix in my Thesis.

The next major step is to run a few benchmark problems through my code to make certain I am seeing the desired output. Then, move into working again with the KAT7 data.

I am excited to get back to literature review, my thesis, and not programming every day :)

Thanks for all the support this past 3 months!

kai

I have been coding for nearly three months, with but a ten days break. I love this time and space. Singularly focused. Only one task in life. I leave my phone at home, launch my development Virtual Machine (no email, no Facebook), and dive in.

My machine learning (a subset of Artificial Intelligence) code “came to life” about ten days ago when it first converged on a solution. That was an incredible feeling. Almost there! Just one more type of mutation to complete. All tools are built but this one. Then it should just work.

While I set out to build a script to work with SKA KAT7 data, my tendency toward building things BIG kicked in and my GP instead became a platform. As a result of my approach, I inadvertently developed a pointer system for managing Numpy arrays. Each column in an axis-1 array is treated as a unique object with parent/child relations to others in the array.

If I had known how hard this would be, I would have looked for alternatives or considered a different plan. But sometimes it is best to just go into a battle blind, look back, and realize how far you have come.