Large-scale comparison of machine learning methods for drug target prediction on ChEMBL



Abstract:

Deep learning is currently the most successful machine learning technique in a wide range of application areas and has recently been applied successfully in drug discovery research to predict potential drug targets and to screen for active molecules.

However, due to
(1) the lack of large-scale studies,
(2) the compound series bias that is characteristic of drug discovery datasets and
(3) the hyperparameter selection bias that comes with the high number of potential deep learning architectures,
it remains unclear whether deep learning can indeed outperform existing computational methods in drug discovery tasks.

We therefore assessed the performance of several deep learning methods on a large-scale drug discovery dataset and compared the results with those of other machine learning and target prediction methods. To avoid potential biases from hyperparameter selection or compound series, we used a nested cluster-cross-validation strategy.

We found
(1) that deep learning methods significantly outperform all competing methods and
(2) that the predictive performance of deep learning is in many cases comparable to that of tests performed in wet labs (i.e., in vitro assays).



BibTeX:

@Article{bib:Mayr2018,
  author="Mayr, Andreas and Klambauer, G{\"u}nter and Unterthiner, Thomas and Steijaert, Marvin and Wegner, J{\"o}rg K. and Ceulemans, Hugo and Clevert, Djork-Arn{\'e} and Hochreiter, Sepp",
  title={{Large-scale comparison of machine learning methods for drug target prediction on ChEMBL}},
  journal="Chem. Sci.",
  year="2018",
  volume="9",
  issue="24",
  pages="5441-5451"
}




Publication
Electronic Supplementary Information
Technical Appendix and Partial Reanalysis

Supplementary_Table_S10.csv
Supplementary_Table_S11.csv
Supplementary_Table_S12.csv
Supplementary_Table_S13.csv
Supplementary_Table_S14.csv
Supplementary_Table_S15.csv



Benchmark Dataset

Compounds

ChEMBL20-SDF
ChEMBL20-MySQL


Static Features (MD5: b8d29262e7d1069052fdf2078e44e6d8)
Semisparse Features (MD5: 0458b111e5bf47b11d5fe3edb44fcd7f)
ECFP6 Features (MD5: f9bac94a3738f8e3c20ad4248832a767)
DFS8 Features (MD5: c26a01d4dbd9db7477f41b87ebfb3361)
Toxicophore Features (MD5: f293b77f62ab6a2dc910cd570b2b4c46)

ECFP4 Features (MD5: d09fca4b2a05a4aeed64375d49124f04, used for clustering)


Training

Sample Map: SampleIndex to ChEMBLId (MD5: df7e773de4ce0272d8ed8207c4ef6a6f, index starting at 0)
Folds (MD5: 7187541a8706bdfbc32d53c6f936266c, columns: SampleIndex, Fold | Folds: 0, 1, 2)

SQL Query Result (MD5: 77789163dd0e75a0fb66c3b703bc0570, Extraction of Data from ChEMBL database with SQL query)
Application of Protocol for Activity Call (MD5: 7b2becf6f52d9f9798631e6529378fc1, columns: Activity, SampleIndex, Target | double entries may exist)

Meaning of Activity:
Activity 0: garbage measurement
Activity 1: inactive
Activity 2: indeterminate
Activity 3: active
Activity 10: weak garbage measurement
Activity 11: weakly inactive
Activity 13: weakely active


Used Assays

tocompute.info (MD5: e5ee2c1c729f1af7190e1d220241a780)





License of our codes: LICENSE


Preprocessed data:

Preprocessed data in C/C++ format and Python format is available here.



C/C++ and Python Pipeline:

The individual steps are given in commdands.sh. More details on these steps are described below.



Code to produce Benchmark Dataset:

available at GitHub
and at git.bioinf.jku.at


ECFP4, ECFP6, DFS8

created with jCompoundMapper (source code [r55]) with 2 files exchanged:


Static (dense) Features:


Semisparse features

  1. MACCS:
  2. PubChem:
  3. SHED, CATS2D:
  4. RDKit Topological Fingerprints (daylight):


Training Labels


Clustering ChEMBL


Considered Targets (=Assays)




Code to reproduce results:

available at GitHub

C/C++ Pipeline


Python Pipeline


Data analysis

Data analysis scripts are available here.

aucPerformancesPerMethod.zip

Contact: Andreas Mayr (mayr@bioinf.jku.at)
Last Update: July, 5th, 2018