When I encounter a system that has already been in development or production for a while, the first sign of real trouble is always a dirty log. You know what I’m talking about. When clicking a single link on a normal flow on a web page results in a deluge of messages in the only log that the system provides. Too much logging is as useless as none at all.
If your systems are like mine, when your job is done, someone else’s job is just starting. After the system has been developed, it will hopefully live a long and prosperous life serving customers. If you’re lucky. How do you know if something goes wrong when the system is in production, and how will you deal with it?
Maybe someone monitors your system for you, or maybe you will monitor it yourself. Either way, what the system logs is going to be critical to know when you want to be woken up in the middle of the night.
For many systems, the first indication that something is wrong is because a log message of the correct level has been written to a log. Mostly, this will be the error log. So do yourself a favor: Make sure from day one that if something is logged the error log, you’re willing to have someone call and wake you in the middle of the night about it. If you can simulate load on your system during system testing, looking at a well-kept error log is also a good first indication that your system is reasonably robust. Or a first warning if it’s not.
For my systems, the best indication that everything is all right, is that the messages at a lower priority are ticking along happily. I want about one INFO-level log message for every significant event for the application.
A cluttered log is an indication that the system will be hard to control once it reaches production. Like with all problems: If you don’t expect anything to show up in the error log, it will be much easier to know what to do when something does show up.