7 Conclusion

When I decided to develop the fann library, I had two reasons for developing it, the first reason was that I wanted to have a fast ANN library which could be used on our robot [Nissen et al., 2003]. The second reason was that I wanted to have full control of a fast ANN library, which could be used in my master thesis. As the project grew, a third reason became more and more evident, I wanted to make a library which would be used by other developers.

These three reasons for developing the fann library had different requirements. If the library should be used on the robot, it should be fast on systems with no floating point processor. If the library should be used in my master thesis, it should be fast on standard workstation machines and it should be very versatile to allow for new functionality to be implemented. If the library should be used by others, it should be both fast, versatile and easy to use.

I knew that it would not be easy to develop a fixed point version of the library, but I also knew that it could be done. During the development of the library several different methods of avoiding integer overflow was suggested (see section 4.4), but I think the method I finally chose was the most elegant. The fixed point fann library was a huge success in all the benchmarks, it was accurate in the quality benchmarks and it was more than 40 times faster than all the other libraries in the performance benchmark on the iPAQ.

The aim for the floating point library, was to be very fast while still allowing both fully and sparse connected ANNs to function. This proved a bigger challenge than first expected and my first implementation did not meet the demands (see section 4.2). The second implementation did however, meet the demands although it is only faster than the lwnn library in some situations. This shows, that although the library is fast, there is still room for improvement.

If the fann library should be used by other developers, speed is not the only requirement. Just as important requirements are, that the library should be easy to use and install, well documented and versatile. The fann library is clearly easy to use, as shown in figure 9 and figure 11. It is also versatile as shown in section 5.3, but what about, easy to install and well documented? No polished install system accompanies the library, hence installation is not easy, still it should not be too difficult to install the library. The user's guide documents how the library should be used, but it does not include a complete reference manual. Appendix B.1.1 fann.h can function as a reference manual, but it is not an ideal solution. In conclusion, the library could easily be used by other developers, but some work still needs to be done in order to make it appealing to a broad audience.

It is sometimes argued that performance engineering is not a real computer science, because new algorithms are not created. I do however not share this opinion and will now give a short example to illustrate the importance of performance engineering. On the AMD Athlon the jneural library uses 509.138 nanoseconds per connection in a fully connected ANN with four layers and 128 neurons in each, on the iPAQ the fixed point fann library uses 188.134 nanoseconds per connection on a similar ANN. This is 2.7 times faster than the jneural library and although it is probably possible to buy a computer, which will execute the jneural library just as fast as the fixed point library on the iPAQ, then it would not be possible to fit this computer into your pocket. This clearly illustrates that sometimes performance engineering is the only choice.

I am most satisfied with the result of this project, the requirements for the library where met and the performance of the library exceeded my expectations. It has been a fun challenge to develop the fann library and I look forward to releasing it on SourceForge.net.

7.1 Future Work

After the library has been released on SourceForge.net, a few changes will be made to make installation of the library easier, but numerous other additions could be made to the library. In this section I will discuss some of these additions.

Many different algorithms can be used for training ANNs, it would be nice to be able to choose from some of these algorithms in the fann library. As a first addition it would be nice have a better gradient descent algorithm like e.g. quickprop [Fahlman, 1988] or RPROP [Riedmiller and Braun, 1993], but later it would be nice to apply more general purpose optimization techniques like e.g. simulated annealing (perhaps this algorithm would do a better job on the thyroid problem).

At this point only two activation functions exists in the fann library (sigmoid and threshold). More activation functions could easily be added, like e.g. the hyperbolic tangent. This activation function has an output in the range of -1 to 1, while the other two activation functions has an output in the range of 0 to 1. Tests have shown that coding binary input as -1 and 1 is better than coding it as 0 and 1 [Sarle, 2002], which in effect would also make an ANN work better if the activation function used for the hidden layers gave an output in the range of -1 to 1. Since the sigmoid function can be altered to give an output in the range of -1 to 1, by the following equation: $y(x) = 2 \times sigmoid(x)-1$, the range of the output from the activation functions could be a separate parameter, which is not determined by the type of activation function.

The cost of calculating the activation functions could be optimized. This could either be done by using the stepwise linear activation function as used in the fixed point library, or by using a table lookup as in the lwnn library.

It seems like fully connected ANNs are usually faster than sparse connected ANNs, it would however still be a good idea to add an optimal brain damage algorithm [LeCun et al., 1990] which could remove unused connections.

If the fann library should be used in developing a learning game bot for Quake III Arena, a reinforcement learning framework should be developed. This framework could either be a part of the fann library, or it could be a separate library which depend on the fann library for ANN support. I think developing the reinforcement learning framework as an independent library would be a good idea, to avoid confusion of what the primary objectives of the fann library are.

All of these additions add new functionality to the library, but other additions, which do not add functionality but rather improve the existing functionality, could also be proposed. Some of these additions could be:

• Add a C++ wrapper to allow a more object oriented approach, to using the library.
• Rewrite the inner loop the execution function in assembler (perhaps with use of the MMX instruction set), for extra performance.
• Add the possibility of more precision to the fixed point library, by allowing the users to chose the long int type as internal representation.
• The layer sizes are given as parameters to the fann_create function, in effect making the number of layers a parameter which should be known at compile time. It would be nice to add another create function which do not have these limitations.

Hopefully many of these additions will be implemented, I will try to implement some of them before I start my master thesis. During my master thesis I will probably implement some of the remaining additions and I will most likely also implement additions not mentioned here.