You probably need to change 2 things:
1) on your model query (before load action on the drawer) set it to be ‘Get more - merge in new rows with old’ instead of ‘Standard - completely replace data.’ This will keep both drawer data sets.
2) To get the correct activities to show for the contact once you have queried more than once, you need to go into your drawer and on your table, set a context condition so the WhoID of the activity is = Id of the contact in context.
Give that a try and let me know if you need some screen shots.
