The impact of digital technologies on the production and consumption of moving images 2007-2016
4. Higher dynamic range research 2010 - 2016
|
As High Definition Video referred to standard definition as its base, so Higher Dynamic Range Video also referred to standard dynamic range as its base. The human eye/brain system has a 15 orders of magnitude system of vision and interpretation of that vision. To understand what that means the reader needs understand that it only has access to 5 orders of magnitude at any one time and that is what you are using now whatever the lighting environment you are in to read this. |
|
- In a starlit landscape those 5 orders of magnitude can slip down to the most sensitive part of the 15 orders to the level of being able to discern one photon
- Then in desert sun it can rise upwards from the most sensitive, through the medium range to the top 5 orders of magnitude which copes with highly illuminated arenas of sight.
- The average displays available in 2007 could represent about 2 orders of magnitude. So standard dynamic range referred to what a normal vacuum tube television could represent (which was far an above early LCD screens which were frankly abysmal) and an ideal HDR display could represent 5 orders of magnitude.
Since 2010, UoB had a prototype display that could reveal 3 – 4 orders of magnitude (peak luminance of 4,000 cd/m), but there was an inability to capture images which promised a greater veracity of capture and display through increases in representation of tonal range. In pursuit of creating the first HDR moving images that were not based upon computer graphics, myself and Dr Marc Price of BBC R&D (a visiting BBC Senior Engineer and research fellow at UoB),computed that we needed to capture 12 gigabytes of image data per second to deliver that image. To understand what that means: the average hard drives of the time could write about 35 MB per second - we were attempting to write data some four thousand two hundred times faster.
We then constructed procedures to capture and display HDR images on UoB’s HDR screen. We staged experiments including the first long-form HDR project which was knowingly entitled ‘The Human Condition’ which was 30 minutes in duration (ironically we achieved display of this work by 2014 after many successive failures to establish the necessary algorithms for transitioning the separate parts of the various signals into a true HDR image). Previous samples of HDR could be measured only as a still image, but in working with Experimental Psychology at UoB, their immersion tests required ‘sustained and compelling footage’ which was of course difficult to achieve on a budget thousands of times lower than Hollywood (in later Hollywood projects such as ‘Tomorrowland’ 2015 - the HDR process involved was a compromise on the potential of a true HDR display for display in cinemas. Budget: an estimated 190 million dollars).
By trying to emulate the characteristics of the eye/brain system I started to fully realise that the information the eye gathers and is then interpreted by the brain - so we should now view the activities of the eye/brain system as interconnected – what’s more, it was no longer sufficient to evaluate what was being looked at, but that we should consider the specificities of the act of the human gaze. My colleague Marc Price then designed new coding procedures to process heavy data loads automatically utilising various algorithms he had designed and some professionally available software. Having completed our practical tests we then co-authored a BBC White paper that described a viable method of capturing and displaying HDR images. We then gave our results at the International Broadcasting Convention in Amsterdam (2012) 9.
Key Research Output at the time (co-authored article)
The new knowledge that emerged during this period of research was disseminated within academic journals and later in 2015 I heard from members of the Science and Technology Committee of the Academy of Motion Pictures Arts and Science (AMPAS10) as well as officers of the American Society of Cinematographers (ASC11) that my research work had attracted their notice such that our propositions were used in their test research. In 2015 and 2016 respectively I was invited to speak directly with these two organisations on the subject of HDR research.
I then left University of Bristol in 2013 to more fully explore my research at the University of the West of England as a professor of Cinematography and Director of the Centre for Moving Image Research.
Collaboration and Insight: Image Resolution and Perception 2013 – 2016
I took up the role of Director of the Centre for Moving Image Research at UWE in March 2013 (CMIR). I conceived of CMIR as having the following aims:
With the first aim I began engaging partners who would also become visiting academics – for instance Roberto Schaefer ASC, AIC who had recently shot and lit Quantum of Solace and various other notable movies – Schaefer had shown interest in exchanging knowledge when I spoke with him on the prime system of knowledge exchange within professional cinematography: The Cinematography Mailing List (CML). Schaefer also introduced me to Frederick Goodich ASC the Sergeant at Arms of the ASC and Schaefer later took this role after Goodich passed away in 2017. With CML, a worldwide knowledge exchange forum in all elements of the practice of film and video (which would come to be known by 2007, as Digital or Data Cinematography), Geoff Boyle FKBS who had started it in the 90’s, was a proponent of rigourous testing of manufacturers claims and had become a well respected contributor at internationally recognised professional and trade forums such as NAB in Los Vegas and IBC in Amsterdam. Amongst others, both became visiting professors at CMIR and I began the process of engaging with industry in earnest.
With the second aim, as mentioned previously Higher Frame Rate capture (HFR) together with HR and HDR represents the eventual goal but of course we could not persist with HFR without much greater advances in data throughput. It is possible to achieve very high frame rates up to a certain resolution of image – but the swap off is more speed equals less resolution as much as more dynamic range would also add higher data throughput requirements. There are also lines of enquiry possible to fluctuate the speed of capture and display such that emotional moods are enhanced via the fluctuation of the display of the frame rate of the image – there is no reason any more with the advent of current technology to hold true to prior technical requirements (i.e. sprocket holes ripping or electron bean whiplash). The demonstration of the hyper real is also an issue as suspension of disbelief could become more difficult with greater veracity in displaying the real world. To truly explore the area of HFR we need something we do not yet have which is a flexible response from the display mechanism such that it follows frame rates demanded of it within the editing process such that new moving image grammar may be invented that accords with frame rates.
I next developed a set of camera and lens tests with CML12, in 2014, thus formalizing professional instrument testing to integrate the practice and the theory of cinematography for professionals and academics. I also brought students from BA, MA and PhD levels together such that the community of understanding was both vertical and horizontal (I was thus exploring the possibilities and principles behind the Immersive Learning Environment). Within a single discipline environment one of my research questions became:
And the subsidiary questions arose:
CMIR and CML published results on the tests at industry events at NAB in Las Vegas and IBC in Amsterdam and on the Cinematographers Mailing list13 and on CMIR’s website for academic colleagues.
By September 2014 I had formulated protocols for an Advanced Innovation Laboratory that took place during the Encounters Short Film Festival in Bristol14. Because we were now assembling professionals, academics and students from different disciplines together, my question next became:
I maintained the intention to create a breakthrough in HDR display via such a laboratory or workshop – such that by the end of a concentrated research period of one week, we would then reveal what we had discovered immediately, in an intelligible way, to a series of 25 person audiences.
Taking Research Results to the International Artistic Research Community
In parallel with the knowledge exchange research, I had decided that I must also bring the results of the HDR research to the International Symposium for Electronic Art (ISEA), which is a yearly gathering of the worldwide community of artists, scientists and academics who deal with digital innovation within art. Subsequently I gave papers at the next 3 iterations, chairing panels in Sydney, Albuquerque and Vancouver from which various articles were derived15. This allowed further and deeper connections around the issue of how knowledge is transferred between artists, academics and professionals.
A step-change in research dissemination occurred when I received a request to speak to the Science and Technology Committee of the Academy of Motion Pictures, Arts and Science in Los Angeles – AMPAS - (June 2015). Here I communicated our research successes, which paralleled and substantiated those of AMPAS itself. This research exchange galvanized the meeting such that committee members proposed that we maintain the channel of comparison of research for the benefit of developing an improved and rigorously robust research process within HDR. Given that this was happening within a highly secretive industry, where representatives of closed research environments were agreeing to open-source their findings, a significant inroad to the centre of the industry for academia had been achieved.
Between the inception of CMIR in 2013 and its thrust towards deeper engagement between academia and industry, I had maintained my own experimental art practice and augmented my research portfolios. To gaze consciously and reflect whilst creating research works and then evaluate research behaviour is a step beyond that of the artist. It is a position that only a researcher/artist can occupy. This becomes even more important now, as emerging technologies such as augmented, virtual and mixed realities research demand a new research framework to assess the ‘internal’ environment of experience they now evoke, the foundations of which can now be based upon the research strategies I originated.
I will further describe the entirety of this development in Portfolio 4
We then constructed procedures to capture and display HDR images on UoB’s HDR screen. We staged experiments including the first long-form HDR project which was knowingly entitled ‘The Human Condition’ which was 30 minutes in duration (ironically we achieved display of this work by 2014 after many successive failures to establish the necessary algorithms for transitioning the separate parts of the various signals into a true HDR image). Previous samples of HDR could be measured only as a still image, but in working with Experimental Psychology at UoB, their immersion tests required ‘sustained and compelling footage’ which was of course difficult to achieve on a budget thousands of times lower than Hollywood (in later Hollywood projects such as ‘Tomorrowland’ 2015 - the HDR process involved was a compromise on the potential of a true HDR display for display in cinemas. Budget: an estimated 190 million dollars).
By trying to emulate the characteristics of the eye/brain system I started to fully realise that the information the eye gathers and is then interpreted by the brain - so we should now view the activities of the eye/brain system as interconnected – what’s more, it was no longer sufficient to evaluate what was being looked at, but that we should consider the specificities of the act of the human gaze. My colleague Marc Price then designed new coding procedures to process heavy data loads automatically utilising various algorithms he had designed and some professionally available software. Having completed our practical tests we then co-authored a BBC White paper that described a viable method of capturing and displaying HDR images. We then gave our results at the International Broadcasting Convention in Amsterdam (2012) 9.
Key Research Output at the time (co-authored article)
- ‘The Production of High Dynamic Range Video’. Authors: Marc Price (1 BBC R&D), Prof David Bull (2 University of Bristol), Terry Flaxton (3 University of the West of England) Stephen Hinde (2 University of Bristol R&D), Richard Salmon (1 BBC R&D), Alia Sheikh (1 BBC R&D), Graham Thomas (1BBC R&D), Dr Aaron Zhang (2University of Bristol): 1 - BBC R&D, Wood Lane, London, W12 7SB, UK 2 - Bristol Vision Institute, University of Bristol, Bristol, BS8 1TH, UK, 3 - University of the West of England, Bristol, BS16 1QY, UK http://www.visualfields.co.uk/HdrIBC2013v7.pdf
The new knowledge that emerged during this period of research was disseminated within academic journals and later in 2015 I heard from members of the Science and Technology Committee of the Academy of Motion Pictures Arts and Science (AMPAS10) as well as officers of the American Society of Cinematographers (ASC11) that my research work had attracted their notice such that our propositions were used in their test research. In 2015 and 2016 respectively I was invited to speak directly with these two organisations on the subject of HDR research.
I then left University of Bristol in 2013 to more fully explore my research at the University of the West of England as a professor of Cinematography and Director of the Centre for Moving Image Research.
Collaboration and Insight: Image Resolution and Perception 2013 – 2016
I took up the role of Director of the Centre for Moving Image Research at UWE in March 2013 (CMIR). I conceived of CMIR as having the following aims:
- Gather the best proponents of industry to collaborate to build new strategies to advance research with those best placed proponents within the subject area from Academia
- To research the technologies of HR, HDR and HFR images in combination, to understand their affects on an audience.
With the first aim I began engaging partners who would also become visiting academics – for instance Roberto Schaefer ASC, AIC who had recently shot and lit Quantum of Solace and various other notable movies – Schaefer had shown interest in exchanging knowledge when I spoke with him on the prime system of knowledge exchange within professional cinematography: The Cinematography Mailing List (CML). Schaefer also introduced me to Frederick Goodich ASC the Sergeant at Arms of the ASC and Schaefer later took this role after Goodich passed away in 2017. With CML, a worldwide knowledge exchange forum in all elements of the practice of film and video (which would come to be known by 2007, as Digital or Data Cinematography), Geoff Boyle FKBS who had started it in the 90’s, was a proponent of rigourous testing of manufacturers claims and had become a well respected contributor at internationally recognised professional and trade forums such as NAB in Los Vegas and IBC in Amsterdam. Amongst others, both became visiting professors at CMIR and I began the process of engaging with industry in earnest.
With the second aim, as mentioned previously Higher Frame Rate capture (HFR) together with HR and HDR represents the eventual goal but of course we could not persist with HFR without much greater advances in data throughput. It is possible to achieve very high frame rates up to a certain resolution of image – but the swap off is more speed equals less resolution as much as more dynamic range would also add higher data throughput requirements. There are also lines of enquiry possible to fluctuate the speed of capture and display such that emotional moods are enhanced via the fluctuation of the display of the frame rate of the image – there is no reason any more with the advent of current technology to hold true to prior technical requirements (i.e. sprocket holes ripping or electron bean whiplash). The demonstration of the hyper real is also an issue as suspension of disbelief could become more difficult with greater veracity in displaying the real world. To truly explore the area of HFR we need something we do not yet have which is a flexible response from the display mechanism such that it follows frame rates demanded of it within the editing process such that new moving image grammar may be invented that accords with frame rates.
I next developed a set of camera and lens tests with CML12, in 2014, thus formalizing professional instrument testing to integrate the practice and the theory of cinematography for professionals and academics. I also brought students from BA, MA and PhD levels together such that the community of understanding was both vertical and horizontal (I was thus exploring the possibilities and principles behind the Immersive Learning Environment). Within a single discipline environment one of my research questions became:
- ‘could it be the case that the newly initiated might contribute innovatory ideas rather than simply receiving teaching within normal terms from people who knew more (either professionals or academics)?’
And the subsidiary questions arose:
- Could the professional inform the academic in their own areas and vice versa?
- And importantly: could an Immersive Learning Environment be a place in which new knowledge could be formulated?
CMIR and CML published results on the tests at industry events at NAB in Las Vegas and IBC in Amsterdam and on the Cinematographers Mailing list13 and on CMIR’s website for academic colleagues.
By September 2014 I had formulated protocols for an Advanced Innovation Laboratory that took place during the Encounters Short Film Festival in Bristol14. Because we were now assembling professionals, academics and students from different disciplines together, my question next became:
- ‘could the extra ingredients offered by different ways of thinking garnered from the protocols of the differing individual disciplines, create more radical innovatory behaviour?’
I maintained the intention to create a breakthrough in HDR display via such a laboratory or workshop – such that by the end of a concentrated research period of one week, we would then reveal what we had discovered immediately, in an intelligible way, to a series of 25 person audiences.
Taking Research Results to the International Artistic Research Community
In parallel with the knowledge exchange research, I had decided that I must also bring the results of the HDR research to the International Symposium for Electronic Art (ISEA), which is a yearly gathering of the worldwide community of artists, scientists and academics who deal with digital innovation within art. Subsequently I gave papers at the next 3 iterations, chairing panels in Sydney, Albuquerque and Vancouver from which various articles were derived15. This allowed further and deeper connections around the issue of how knowledge is transferred between artists, academics and professionals.
A step-change in research dissemination occurred when I received a request to speak to the Science and Technology Committee of the Academy of Motion Pictures, Arts and Science in Los Angeles – AMPAS - (June 2015). Here I communicated our research successes, which paralleled and substantiated those of AMPAS itself. This research exchange galvanized the meeting such that committee members proposed that we maintain the channel of comparison of research for the benefit of developing an improved and rigorously robust research process within HDR. Given that this was happening within a highly secretive industry, where representatives of closed research environments were agreeing to open-source their findings, a significant inroad to the centre of the industry for academia had been achieved.
Between the inception of CMIR in 2013 and its thrust towards deeper engagement between academia and industry, I had maintained my own experimental art practice and augmented my research portfolios. To gaze consciously and reflect whilst creating research works and then evaluate research behaviour is a step beyond that of the artist. It is a position that only a researcher/artist can occupy. This becomes even more important now, as emerging technologies such as augmented, virtual and mixed realities research demand a new research framework to assess the ‘internal’ environment of experience they now evoke, the foundations of which can now be based upon the research strategies I originated.
I will further describe the entirety of this development in Portfolio 4
Moving Image Arts Research is concerned with exploring the histories, theories, technologies, cultures and politics of moving image art production, interaction and reception.
|
|
Interactive Region Hot Spots
small experiments in connectivity and how drama or documentary might be enabled |