Molecular markers (SSR, SNP and DArT) and agronomical traits have been used in the world’s largest olive (Olea europaea L.) germplasm collection (IFAPA, Centre Alameda del Obispo, Cordoba, Spain) to study the patterns of genetic diversity and underlying genetic structure among 361 olive accessions. In addition the marker data were used to construct a set of core collections by means of two different algorithms (MSTRAT and PowerCore) based on M (maximization) strategy. Our results confirm that the germplasm collection is a useful source of genetically diverse material. We also found that geographical origin is an important factor structuring genetic diversity in olive. Subsets of 18, 27, 36, 45 and 68 olive accessions, representing respectively 5%, 7.5%, 10%, 12.5% and 19% of the whole germplasm collection, were selected based on the information obtained by all the data set as well as each marker type considered individually. According to our results, the core collections that represent between 19% and 10% of the total collection size could be considered as optimal to retain the bulk of the genetic diversity found in this collection. Due to its high efficiency at capturing all the alleles/traits states found in the whole collection, the core size of 68 accessions could be of special interest for genetic conservation applications in olive. The high average genetic distance and diversity and the almost equal representation of accessions from different geographical regions indicate that the core size of 36 accessions, could be the working collection for olive breeders.