We present the Four-Leaf Clover (FLC) dataset, a new experimental framework for studying fine-grained object localization problems. We built the FLC dataset with the contribution of trained hobbyists, who were assigned the task of spotting four-leaf clovers on a fixed geographical extension over two clover seasons, one season for the train set and another for the test set. We then annotated each object instance for the tasks of object detection, semantic segmentation, instance segmentation, object parsing and semantic boundary detection. Our dataset is composed of more than 100,000 images, containing 2,151 carefully annotated clover instances of four, five or six leaves. The FLC dataset is extremely challenging and adapted to fine-grained object localization problems due to its small inter-class variance and its very large intra-class variation. We perform extensive experiments with state-of-the-art methods in order to establish strong baselines for each of the tasks.