Caterpillars inside my lemons!

I uncovered my lemon tree this afternoon as we are due to have a few mild nights and there were some leaves bound together with white flossy stuff.
On closer examination I found some of the lemons with holes in.
I took one in the house and cut it open and there was a tiny green caterpillar which soon crawled out once disturbed.
I checked all the lemons and have had to remove about 10 altogether with holes in. *See photo. You can actually see a small green caterpillar on one of the lemons.
I also found a couple that had turned into the chrysalis stage under leaves.
The tree is in my greenhouse for winter and the pot is well wrapped and it gets a large thick fleece bag over it on frosty nights. The lemons are all small and immature. I’ve probably got about 12-15 left on the tree so no real harm done as there were probably too many for a small tree - about three feet tall in a large pot.
What sort of caterpillar would this be?
I’m going to keep checking it over regularly, I don’t really want to spray it if I can help it.
I do have some neem oil if needed.
I’ve had the tree about 18 months and overwintered it the same way last year and it spends spring and summer on the patio.

 

Looks like some sort of tortrix moth. they spin silk normally joining leaves together so they can munch in peace without risk of predation. Open up a lemon to see if they've gone past the pith as I would've thought the fruit itself was too acrid for them....could be wrong

 

It`s hard to know. I cut one lemon open but was trying to be gentle as I didn`t want to cut whatever beastie was inside in half. As I was cutting and peeling it apart an alarmed caterpillar crawled out.
There were definitely a few binding leaves together and hiding away to turn into moths. The pith on a small lemon is quite thick. So possibly eating that before they go to the next stage. Obviously nice and warm inside a lemon inside a thick fleece bag in a Yorkshire winter. As long as my lemon tree isn`t in danger I can afford to lose 10 lemons.

 

I'd be inclined to just go over the bush and kill all the caterpillars you find, I usually just squeeze any leaves that are spun together , or you could just cut them off if there are not too many.

You did the right thing taking off the affected fruit, best to throw that in the rubbish bin.

 

To be fair, no one wants to cut into a lemon for their G&T and have a caterpillar float to the top of the glass

 

maybe they have timed it right , been in since you covered up your plant hatched out in the warmth of fleece even a fruity appetizer .