Call For Submissions to ICML 2008 Workshop The Third Workshop on Evaluation Methods for Machine Learning William Klement, Chris Drummond, Nathalie Japkowicz and Sofus Macskassy Description of the topic ======================== The workshop would be the third in a series, the previous ones having taken place at AAAI over the past two years. Our continuing goal is to encourage debate within the machine learning community into how we experimentally evaluate new algorithms. The earlier workshops were successful in that they began the process of presentation, and discussion, of new ideas for evaluation. However, they did not raise all the high-level questions we believe must be addressed by the community. For this reason, we decided to call on the machine learning community to participate in the discussion to address the important, high-level, issues surrounding machine learning evaluation by soliciting position papers rather than research papers. We invite position papers addressing the list of the issues raised in previous workshops and other ones which we feel warrant further discussion. We also suggest a possible range of positions that might be taken on these issues: * Should we substantially change how evaluation is performed in machine learning? o Yes: There are some serious problems with how we do things, making our experimental results of questionable value. o No: We largely have the methodology worked out. A new metric and a few more data sets should solve any outstanding problems. * Is evaluation the fundamental role of experiments? o Yes: Careful experimental evaluation is critical for separating good ideas from bad ones. o No: Experiments should be used to explore ideas, discover relationships, compare alternatives, testing hypotheses is only a small part of the process. * Should we use one well chosen, community wide, evaluation measure (e.g., Accuracy, AUC, F-measure)? o Yes: It gives a clear and definitive answer to which algorithm is the best. o No: There are many different, and important, properties of algorithms. More than a few measures are needed to capture them all. * Are statistical tests critical to evaluation? o Yes: They are very critical, but we need a deeper understanding of their meaning. o No: Their value is considerably overstated, they are of limited merit. * Are the UCI data sets sufficient for evaluation? o Yes: They represent many practical problems and new additions, focusing on new domains, are added all time o No: They are not a representative sample of possible problems. Many more data sets, plus artificial data, are needed to explore all variations that may occur. This list certainly does not capture all the issues worthy of discussion nor the possible positions. We expect, and very much encourage, position papers raising other issues that members of the machine learning community think important. Proposed Format =============== Authors of submissions to the workshop will be invited to present their ideas in 4 sessions of panel discussions in a round-table format rather than a classical paper-presentation format. Authors will have the chance to present their ideas to the panel and to participate in the debate. Submission ========== Authors are invited to submit papers on the issues listed above or other related positions. Only position reports will be considered for this workshop. To promote a lively event, with plenty of discussion, the organizers are very interested in papers taking strong positions on the above topics. Workshop papers should be at least 1 page long and should not exceed 4 pages using the ICML'08 Style. Submissions should be made electronically in PDF or Postscript format and should be sent (no later than May 1, 2008) by email to William Klement: william.klement@gmail.com Organizers ========== Chris Drummond NRC Institute for Information Technology, Canada Email: chris.drummond@nrc-cnrc.gc.ca William Klement (main contact) School of Information Technology and Engineering, University of Ottawa, 800 King Edward Ave., room 5-105, Ottawa, Ontario, K1N 6N5 Canada Telephone: +1 (613) 562 5800 Extension 6699 Fax.: +1 (613) 562 5664 Email: william.klement@gmail.com Nathalie Japkowicz School of Information Technology and Engineering University of Ottawa, Canada Email: nat@site.uottawa.ca Sofus A. Macskassy Fetch Technologies United States of America Email: sofmac@fetch.com Workshop Website ================ http://www.site.uottawa.ca/ICML08WS/