A director and his serial killer star set out to make radical docu-fiction, until differing creative visions cause tensions to arise