A Docker-Based Replicability Study of a Neural Information - PowerPoint PPT Presentation

A Docker-Based Replicability Study of a Neural Information Retrieval Model Nicola Ferro, Stefano Marchesin, Alberto Purpura , Gianmaria Silvello 
 University of Padua, Italy The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

Neural Vector Space Model � w k , . . . , w k + n ∈ d p � d p � d n Minimize Maximize 
 Word n-gram the distance the distance Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Neural Vector Spaces for Unsupervised Information Retrieval. ACM Trans. Inf. Syst. 36, 4, Article 38 (June 2018), 25 pages. DOI: https://doi.org/10.1145/3196826. The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) � 2 July 25, Paris, France

<latexit sha1_base64="ScYDaTIPx7Te3S+eIYKxeM4GIxc=">AF3ictVTPb9MwFM62Bkb41cGRi0U1B1W2WX9wa1QpMFhUDraTVqynHd1prjRI6DVIVcuHAIa78W9z4Q7jdGlX1i5DmnhR4qf3nv1937Njx+csUBD+WlvfyJk3bm7esm7fuXvfn7rQTfwQkloh3jck8cODihngnYU5we+5Ji1+H0yDltJvmjD1QGzBPv1cSnPRePBsygpUO9bc2ftsOHTERKeyEHMs4kh/TJ7bASnsCDp639Fe8bO43EITabelRD21KMOcLpbYN7HFCzdp+0z08AMW3kmkwzHfma9luyBXTQkJXRCiBjyNYQhUYX5os12sZyWf1JDlfH6EVEs5zQgtqPtPjHZ3F1APX7fbTSBDcTneXjljyVo1i8wUD5aqHqudAbdbHVAMQGO1cIflrPIoCyCFRSArC+0Or9f0fO6ma1kWrNpMOVyGj62nOgFzZFhbhVy+nuaraSWakUZOd3vJLJuKwfwv7+cLsASnBpYdlDoFI7VWP/THngkdKlQhOMgOEHQV70IS02I09iyw4D6mJziET3RrsAuDXrR9H6KwbaODMDQk/oVCkyjizMi7AbBxHV0pYvVOLiYS4KrciehGtZ7ERN+qKgZ0DkAPlgeSyAwMmKVF8oh1MJNcARljiYnSV6Klm4AuSl52umW9OaXyu71C40Xajk3jkfHYKBrIqBkN45XRMjoGydm5T7kvua8mNj+b38zvZ6Xra+mch8ZfZv74A1vGzec=</latexit> Results d e t a c i l p e R MAP nDCG@100 P@10 Recall Original 0.150 0.287 0.298 – OSIRRC run 0.142 0.276 0.288 0.616 CPU (run 0) 0.138 0.271 0.285 0.608 GPU (run 0) 0.137 0.265 0.277 0.610 GPU (run 1) 0.138 0.270 0.277 0.607 GPU (run 2) 0.137 0.268 0.270 0.611 Retrieval results on the Robust04 (T) collection computed with the two shared Docker images of NVSM. The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) � 3 July 25, Paris, France

<latexit sha1_base64="betvrDmKt1kp0hCev74OFC2a9ds=">ADfXicjVJPT9swFHeTMaBjWxnHXaxVIqmKMlAlBsaB3bspBWQmqpy3NfWwnEi20GqQj/Fvhk3vgqXzWnTgdtN2rMsv3+/3t+enHGmdK+/1hz3Fcbrze3tutvdt6+e9/Y/XCl0lxS6NKUp/ImJgo4E9DVTHO4ySQJOZwHd9elPHrO5CKpeKHnmbQT8hYsBGjRBvXYLf2ExuJYhgzUWgS5zIWSHvqzOrY1sO8GWniw9lLrDfemkFlhW1oWxoialL3ZPDYHxoHnz7l9zw9P7DdoL0HrVHbxJcia59VNMf/QRO27NpzsEXn/2bpvzu85gWrS9rL8HPdCsMEYjhn/kPGs0yvRS8rgSV0kSVdAaNh2iY0jwBoSknSvUCP9P9gkjNKIdZPcoVZITekjH0jCpIAqpfzLdnhveNZ4hHqTRXaDz3vkQUJFqmsQmMyF6olZjpfNvsV6uR+1+wUSWaxB0UWiUc6xTXK4iHjIJVPOpUQiVzPSK6YRIQrVZ2LoZQrD65XlKvSCL174/bh5/rUaxb6iD6hQxSgU3SOvqEO6iJae3Kw03KOnF/uvZ9RapTq3C7CFL3NPfSk/kPg=</latexit> Runs Similarity GPU (run 0) GPU (run 1) GPU (run 2) CPU GPU (run 0) 1.0 0.025 0.025 0.018 GPU (run 1) 0.025 1.0 0.089 0.014 GPU (run 2) 0.025 0.089 1.0 0.009 CPU 0.018 0.014 0.009 1.0 Kendall's � correlation coe ffi cient values between the runs we τ computed with the NVSM GPU and CPU Docker images considering the top 100 ranked documents in each run. The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) � 4 July 25, Paris, France

<latexit sha1_base64="tr0RPZ4kVOdIXYPbH/vZI3kDgx0=">AAAECXicpVNNb9MwGPYSPkb56tiRAxYVaDu0itska27TJvFxK4Juk5qqclyntZY4ke2gVaFXLvwVLhxAiCv/gBv/BqcLVZuOXXijWM/7vB9+XssO0ohJZVm/twzzxs1bt7fv1O7eu//gYX3n0YlMMkFonyRRIs4CLGnEOO0rpiJ6lgqK4yCip8H5cRE/fU+FZAl/p2YpHcZ4wlnICFaaGu0YT6A2P6ATxnOFgyzCYp6LD4tvXoOlHff68Dl8qdc9kXFo7a96aM1r70Pfh/60ULSs9xW9UEGYvzh6/bbTdJw28ua66P9p31/uUdB200aehyrJdvOgi1C1xbXsSt/1qO3Y7oYaTXc81ynodRUrXkWpHsD2bGdjrHbH67qb015L/1Ps3wGvEuU6tu2ue2Ufn/Lx8i6M6g2rZS0MbgJUggYorTeq//LHCcliyhWJsJQDZKVqmGOhGInovOZnkqaYnOMJHWjIcUzlMF/c5Dl8ppkxDBOhf67ggl2tyHEs5SwOdGaM1VRWYwV5VWyQqbA7zBlPM0U5udwozCKoElg8CzhmghIVzTTARDCtFZIpFpgo/Xhq+hBQdeRNcNJuoU6r/cZuHB6Vx7ENHoOnYA8gcAAOwSvQA31AjI/GZ+Or8c38ZH4xv5s/LlONrbJmF6yZ+fMPnswh/A==</latexit> Final Remarks CPU GPU (run 0) GPU (run 1) GPU (run 2) FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS4-41991 FBIS4-7811 FBIS4-7811 FBIS4-7811 FBIS4-41991 FBIS4-41991 FBIS4-45469 FBIS4-43965 FBIS3-54945 FBIS3-23986 FBIS3-23986 FBIS3-23986 FBIS4-41991 FBIS4-65446 FBIS4-65446 FBIS4-7811 Top 5 documents in the runs computed with the CPU and the GPU. Relevant documents are highlighted in bold. The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) � 5 July 25, Paris, France

Thank you! Alberto Purpura, purpuraa@dei.unipd.it