This paper describes visual interaction mechanisms for image database systems. The typical mechanisms for visual interactions are query by visual example and query by subjective descriptions. The former includes a sketch retrieval function and a similarity retrieval function, while the latter includes a sense retrieval function. We adopt both an image model and a user model to interpret and operate the contents of image data from the user''s viewpoint. The image model describes the graphical features of image data, while the user model reflects the visual perception processes of the user. These models, automatically created by image analysis and statistical learning, are referred to as abstract indexes stored in relational tables. These algorithms are developed on our experimental database system, the TRADEMARK and the ART MUSEUM.