Distributed computing for data intensive research

The GridPP Collaboration is a community of particle physicists and computer scientists based in the United Kingdom and at CERN. Drawing on expertise from nineteen UK institutions, our vision is to create, manage and oversee the evolution of the computing infrastructure needed to maintain the UK’s position as world leaders in particle physics and distributed computing. We do this by using and actively contributing to the development of open source software, applications and middleware needed to power large-scale distributed “high throughput” computing for particle physics and beyond.

Since starting as a research and development project in 2000 it now specifies, runs and manages the UK contribution to the World Wide LHC Computing Grid (WLCG) and the European Grid Infrastructure (EGI) that provides a large-scale computing and data storage facility that is geographically distributed. Recent advances in high-speed networks have allowed us to move from a strictly hierarchical structure to one in which any data set on the WLCG anywhere within the world can be accessed at any time on any compute server.

Brunel hosts one of the UK Tier-2 centres for the WLCG. This is connected to the external wide-area-network at 20 GBit/s. In the last 12 months Brunel Tier-2 has provided over 16 million cpu hours of computational facility and 1400 TBytes of accessible, fast disk storage.

The use of high-throughput distributed computing systems took off around 2000 in specialist fields like particle physics and weather forecasting. It has since spread via astronomy, genomics, medicine, and all aspects of scientific computing, until there is now hardly an academic field that does not make some use of its power. Many of the ideas and techniques pioneered by GridPP and our collaborators, now support the work of researchers from every field of study.

This video is a screen capture made in August 2017 of the EGL application built by the CERN medialab. This application provides a close to real-time (refreshed every ten minutes) visualisation of the WLCG grid activity. It shows graphically information covering data movement for both montecarlo production and tier0 export, and additional information about running jobs on the ATLAS, CMS, LHCb and Alice (jobs only) grids.  Copyright 2018 CERN