CSAR Benchmark Exercise 2011–2012: Evaluation of Results from Docking and Relative Ranking of Blinded Congeneric Series
Open Access
- 10 May 2013
- journal article
- research article
- Published by American Chemical Society (ACS) in Journal of Chemical Information and Modeling
- Vol. 53 (8), 1853-1870
- https://doi.org/10.1021/ci400025f
Abstract
The Community Structure–Activity Resource (CSAR) recently held its first blinded exercise based on data provided by Abbott, Vertex, and colleagues at the University of Michigan, Ann Arbor. A total of 20 research groups submitted results for the benchmark exercise where the goal was to compare different improvements for pose prediction, enrichment, and relative ranking of congeneric series of compounds. The exercise was built around blinded high-quality experimental data from four protein targets: LpxC, Urokinase, Chk1, and Erk2. Pose prediction proved to be the most straightforward task, and most methods were able to successfully reproduce binding poses when the crystal structure employed was co-crystallized with a ligand from the same chemical series. Multiple evaluation metrics were examined, and we found that RMSD and native contact metrics together provide a robust evaluation of the predicted poses. It was notable that most scoring functions underpredicted contacts between the hetero atoms (i.e., N, O, S, etc.) of the protein and ligand. Relative ranking was found to be the most difficult area for the methods, but many of the scoring functions were able to properly identify Urokinase actives from the inactives in the series. Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended. Pregenerated ligand conformations performed better than those that were generated on the fly. Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting. Lastly, for both sampling and ranking scoring functions, the use of the empirical scoring function appeared to trend positively with the RMSD. Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.Keywords
This publication has 63 references indexed in Scilit:
- Surflex-Dock: Docking benchmarks and real-world applicationJournal of Computer-Aided Molecular Design, 2012
- Variability in docking success rates due to dataset preparationJournal of Computer-Aided Molecular Design, 2012
- Species-Specific and Inhibitor-Dependent Conformations of LpxC: Implications for Antibiotic DesignCell Chemical Biology, 2010
- Computer-aided drug-discovery techniques that account for receptor flexibilityCurrent Opinion in Pharmacology, 2010
- Conformer Generation with OMEGA: Algorithm and Validation Using High Quality Structures from the Protein Databank and Cambridge Structural DatabaseJournal of Chemical Information and Modeling, 2010
- Effects of protein conformation in docking: improved pose prediction through protein pocket adaptationJournal of Computer-Aided Molecular Design, 2009
- Recommendations for evaluation of computational methodsJournal of Computer-Aided Molecular Design, 2008
- Customizing scoring functions for dockingJournal of Computer-Aided Molecular Design, 2008
- How to do an evaluation: pitfalls and trapsJournal of Computer-Aided Molecular Design, 2008
- Assessment of programs for ligand binding affinity predictionJournal of Computational Chemistry, 2008