The judges ‘had never read anything quite like it’, says panel chair Roddy Doyle, announcing the Hungarian-British author’s novel as the winner of the £50,000 award