If you look at the modern world, perhaps the closest thing to formulation of goals for an AI is the formulation of KPIs of our companies — our artificial social intelligences. They sometimes go wrong. For example, a company may choose maximization of total time spent on site, rather than the people's time well spent.
The very reason why people use KPIs, is that money is not good enough an abstraction of our goals, which are complex and expressible only in an evolving diversity of assets.
How organizations define KPIs?
People have centuries of experience defining KPIs for their organizations. With a few searches on-line, you may find, that some of the modern abstractions are:
Image credits: Bryan Cable @ Transportation Insight.
However, this is just a start. There is a whole Universe of things with respect to which to define what we want.
Every even tinyest dot in this image, is a galaxy, not to forget the mathematical universes and all potential futures of individual experiences of every one of us...