4 considerations for Machine Learning System in Production
Source:- infoworld.com
Writing a recommendation engine isn’t easy, but at least it’s straightforward. Given a large data set and some symptoms, you can determine what disease a patient might have.
The problem is getting that recommendation engine, written in R or Python, and integrating it with an existing medical records system written in a more traditional language and delivered over the web.
Most of the attempts to do this look a bit like a wonderful mosaic quilt … with a machine learning system Krazy Glued to the end and duct taped to the source database the algorithm will use.
The challenge is that the neural network code will have to be integrated into other code, whether it is a user interface, Web interface, database, or control software for electromechanical embedded systems. The intelligent network code has to be able to integrate into a larger software or hardware/software system.
Considerations for Machine Learning Systems in Production
1. Is it ready from a requirements standpoint to move into production? Assuming you have requirements for your system, have you determined whether you met those requirements? How would you test for it – both independently and as part of a system (and both the initial release and maintenance releases)?
2. Is it giving you the results you expect? While the network has been trained, you need to run both the algorithms and the supporting application code against real life data to ensure that the application as a whole is behaving as expected. More importantly, you might want to keep running tests in production to see if the system is getting better over time.
3. How about performance and scalability? If you’re not testing for performance and scalability, your system has a big question mark at the end of it. We’ve seen systems that hit performance problems with as few as four simultaneous users. If you’re an embedded or real-time system, the system may need guaranteed response times, a real challenge in a functional programming language.
4. Is the algorithm and accompanying application code reliable? That’s a several part statement, including the ability of the algorithm to produce results that reflect the needs of the application. Does it have to be exact? Or is it acceptable that the results vary within a defined range? And what if a few results are clearly inaccurate?
Automate Integration with DevOps
The Krazy Glued piece typically means two different code bases, often in two different version control systems. Deploying the “current version” can lead to debugging, fixing, and rollback issues – but it doesn’t have to. Modern continuous integration servers, like Jenkins, can be configured to automate the build while tracking the exact version of each system. DevOps principles can automate the deployment, replacing a human, prone to error, with an automated script.
Another common DevOps component is monitoring production for performance. Machine learning systems can take that component one step further – adding in a continuous evaluation of the effectiveness of the Neural Net in production. If the system begins to decay, the software can send an alarm. For example, after years of recommendations, Amazon.com once had an error where people considering televangelist Pat Robertson’s book were recommended a sex manual.
DevOps is built to identify deficiencies fast, and to make corrections just as fast – often at the click of a button.
Errors happen, by algorithm, or training, or just integration. When your Machine Learning System is making recommendations that do not compute, rollback at the push of a button might be exactly what you need.