An attractive broad view of vision is that it is an inference problem: we have some measurements, and we wish to determine what caused them, using a mode. There are crucial features that distinguish vision from many other inference problems: firstly, there is an awful lot of data, and secondly, we don’t know which of these data items come from objects — and so help with solving the inference problem — and which do not. For example, it is very di?cult to tell whether a pixel lieson the dalmation in figure 16.1 simply by looking at the pixel.

This problem can be addressed by working with a compact representation of the "interesting" image data that emphasizes the properties that make it "interesting". Obtaining this representation is known as segmentation. It’s hard to see that there could be a comprehensive theory of segmentation, not least because what is interesting and what is not depends on the application.

There is certainly no comprehensive theory of segmentation at time of writing, and the term is used in different ways in different quarters. In this chapter we describe segmentation processes that have no probabilistic interpretation. In the following chapter, we deal with more complex probabilistic algorithms. Segmentation is a broad term, covering a wide variety of problems and of techniques. We have collected a representative set of ideas in this chapter and in chapter ??.

These methods deal with different kinds of data set: some are intended for images, some are intended for video sequences and some are intended to be applied to tokens — placeholders that indicate the presence of an interesting pattern, say a spot or a dot or an edge point (figure 16.1). While superficially these methods may seem quite different, there is a strong similarity amongst them. Each method attempts to obtain a compact representation of its data set using some form of model of similarity (in some cases, one has to look quite hard to spot the model). One natural view of segmentation is that we are attempting to determine which components of a data set naturally "belong together". This is a problem known as clustering; there is a wide literature.