And here on a modern view of the area,
The street the factory was on, Greensite, continued into a series of slums that were demolished to give way to the modern buildings we see today. The site of the factory seems to have been developed long, ago, occupied by the Wordsworth House and Stables, here is a map of 1819,
Here is a plan and sketch from the permit application. Is that arch I found in the street a remnant from one of the depicted ones?
Nigel points out that the stables were bought by George Sandeman, owner of Sandeman Whisky, which erected the three extra stories on the building that ended up being the Royal Enfield factory.