Once we had the process working, we created a series of tests using clips from existing movies to prove out the pipeline and refine the various thresholds. From here, the process was established – for each clip, the Half Mermaid team would review and tag a single instance of each object within the scene, giving it a unique ID. Then, a team of rotoscopers would take the scene and fully track all the objects – exporting the resulting huge pile of data into the game-ready format.
The Half Mermaid team would then review in game using debug tools we created to allow us to quickly jump through every item in a scene, cycling through frames the game code highlighted as being good potential cuts. The rotoscoping would always be reviewed in game to ensure nothing strange had been introduced in compression, and to further verify that the match cut algorithm was picking out ‘good’ frames against the final data.
By creating a data-driven system as described above, we were able to create a mechanic that was rich and player-responsive – it wasn’t reliant on designers tagging connections, or picking out the cuts that they had anticipated. As shipped, IMMORTALITY has over a million possible cuts that it can make – far too many connections for us to have laid down by hand. And yet, the system is capable of creating magic!
When we started prototyping these elements, we set a target of aiming for one cool cut per play session, justifying that players would remember the cool cuts and take that forward with them.
Source: Unity Technologies Blog