Hello ThanushiyahG,
the problem is probably related to the access to the data. I suppose, when accessing a data record, your implementation iterates through the data records. The more records, the longer the time to iterate.
The Vertical List itself limits to cache few visible items. It is not depending on the number of items. Many items should not affect the performance of the list.
The unique peculiarity is when the list is intended to display items with flexible item height. In such case it is inevitable to iterate over the items in order to estimate the position of an item. To achieve best performance, configure the list to calculate with fixed size items.
Best regards
Paul Banach