Remove the list view threshold (5000 by default)
This limit has always been a bit laughable, and is even more so as we develop more client side applications. In SharePoint 2007 we didn't have this limit and were allowed to make our own mistakes. Now that hardware is so much more powerful, we need this limit removed so that we can build enterprise-class applications.
We are continuing to make our large list experiences better, please keep the feedback coming.
Spring 2018 update:
- We now support being able to manually add indexes to lists of any size (increased from lists up to 20,000 items previously).
- Starting with the February release of the Office 365 Excel client, you will be able to export your full list instead of getting cut off part of the way through.
What we are working on now:
- Predictive indexing will start to work for lists larger than 20,000 items so your views will automatically cause the right indexes to be added to your lists.
In our backlog:
- Being able to index/sort/filter by lookup column types (like person, lookup or managed metadata columns) without being throttled.
- Making sure that our REST APIs support querying in ways that will guarantee that the call will not be throttled.
For a general update on large list capabiltiies, the video on myignite.microsoft.com/videos/53861 (focusing on large lists at 42 minutes and 25 seconds) describes some of the changes that we delivered back in the second half of 2017:
- Modern UI now has a lot of support for adding indexes to large lists and libraries on the fly, reducing the number of throttling errors experienced by our users, and some new UI for browsing through items in large lists with our paging model
- SharePoint runs predictive indexing jobs to automatically add indexes as lists get larger based on your view definitions, and updates these indexes when you add/update your views
Looking forward to receiving more of your feedback.
Noah Stahl commented
I don't think the proposal is reasonable. This is as others have pointed out a fundamental blocker that can't realistically be overcome by anticipating a limit of records in a given view. I think this should be top priority to solve. Surely the technical capability and infrastructure is there to make this possible.
San Mattel commented
5000 default limit is really frustrating. Many cases client not ready to take a risk by increasing this limit and looking at other solutions
Russell Gove commented
If The 5,000 item threshold is likely here to stay, then we need to have better tools to manage it. I dont know I have an issue until a user calls and reports an error. At that time it;'s often too late to provide a reasonable solution.
It would be nice if an admin could set the listview threshold to a lower limit (say maybe 4000. Then, when a user calls to report an issue the admin can bump it up to 5000 to get the users up and running, then add indexes, create views, whatever to resolve the issue, and then set it back to the lower limit. This would give us advancred notice and the ability to test if our workarounds actually work
Jim K commented
Microsoft seems to be positioning sharepoint online as a google drive, box, dropbox competitor - i.e. a file server replacement. None of these other services have this ridiculous restriction. It's a deal killer for using sharepoint online as a file server replacement. Which, in turn, makes us want to look at other solutions such as g suite.
The response by the SharePoint Experiences Team seems to reflect an upside down view of the world. Technology is subservient to people, not the opposite. Asking list owners and users to work around, or against, limits not justified by current technology is not reasonable, and neither is expecting users to think like developers.
Balu Dharmarajan commented
This limit is quite low, I already checked the dll codes and sql queries and it looks like not too complicated to fix, please let me know I will tell you how it can be fixed.
At the Virtual SharePoint Summit, on a slide titled 'Transform business process roadmap', MS announced 'By year end: intelligent indexing for large lists'. Please MS don't forget about any column type: https://ktnnsharepoint.wordpress.com/2016/10/22/column-types-list-view-threshold/
Pramod Nathan commented
Remove List view threshold limit of 5000 by taking advantage of paging
The current list view threshold limit of 5000 with it's workarounds are very frustrating. We can all appreciate that we do not want to fetch too many items from the table (the underlying cause of the restriction). BUT paging exists and Microsoft needs to take better advantage of this:
1. The SharePoint interface is page based. Instead of breaking (which is what the solution does with a warning), MS can change the underlying call so it breaks down the call to SQL to fetch data as needed per page/set of pages. When the user navigates to the next page/set of pages - call SQL for the next set of data (e.g. if I am on the current view with 5000+items, I really don't care about about item 5000+1 immediately. BUT I do care that I can manipulate the entire set of items and that I can get to item 5001). So just fetch the first few page worth of items.
2. For sorting and filtering, the call can go to SQL on click to fetch the top X number of items depending on the filter or sort. (e.g. if the page has 5000+items and we want to sort by a column, it can call a SQL statement that does this but sends back only the top 3-4 pages or how many ever are feasible)
3. It will also make the underlying APIs very robust in terms of being able to create multiple calls of data to generate a large data sets that are stored in SharePoint lists and libraries.
the 5,000 is pushing us to pursue other solutions outside SharePoint.
We created a list to hold user's stats; and for stats purposes it is worthless to retrieve chunks of data when you want to know how many selected a, b, or c, or group them by hour.
As a workaround, we connected the list to PowerBI. From there we created a dashboard which actually can work with all the data at the same time. Until now, this workaround worked just fine, but we just got a message saying that dashboard sharing will no longer be part of PowerBI free.
We just want SharePoint to work and not to start thinking on workarounds or 3rd party applications to make it effective and efficient.
Steven Derveaux commented
And another customer bites to dust...
Keep in mind that the search box in document libraries (in the modern experience) is also struggling with this limit. Working with folders can overrule the 5000 items, but the new-autocompleted-search cannot cross this limit. MS Support pointed us to use the Search Center or create our own webparts. Meanless to say: you cannot replace this autocompletion search webpart because you cannot customize the modern view! Aargh!
So what can you do? switch back to Classic Design but then you loose the new copy functionality!
I have no words for this...
This limit of 5000 thousands is preventing us of creating many solutions in Sharepoint.
SharePoint is heading backwards in so many ways. As another contributor said, it's driving business away from SPO - but then, maybe that's what Microsoft want. SharePoint will become a simple (and small) file store). Lots of publicity re huge amounts of storage space, millions of items in lists etc. - except it's effectively unusable.
What makes this limit really unusable is that you cannot create / amend indexed fields. How useless is that!
Or am I being cynical?
The 5000 item threshold is ridiculous when Microsoft's own guidelines say a library can hold 30 million items. How in the world is anyone supposed to come up with indexes and filters to split that into 5000 item chunks and have it be anything but worthless? It's already difficult enough when people have barely more than 5000 and a view using Groupings, and suddenly it stops working after a migration.
Juan Pineiro commented
This has been a major issue for us and driving business off of SPO. It's not just about awareness, indexes or filters. I understand nobody need 5k+ items sent to a page, but after 5k, many setting features no longer work. We have a list with 25k items and you can't add new indexes or change calculated columns on it. Is that a bug or a protective feature?
Arvinda Sudan commented
We are really disappointed with the 5000 limitation as we are online users. Hope Microsoft will find a good solution for this in the future. We also appreciate if someone can suggest us a third party solution to overcome this 5000 limitation.
Russell Bain commented
Admin - "The 5,000 item threshold is likely here to stay. We are aware that it takes awareness on both sides to get past this limitation".
Awareness is not the solution - it is unrealistic to expect the average SharePoint user to develop the technical skills required to operate under this sort of limitation. Even those with strong technical ability are often unable to mange such lists and libraries effectively because finding the right approach often depends on identifying usage patterns and business needs which may not yet be established, or may change significantly over time. Besides, in many cases there is no approach which meets all of the critical business needs effectively.
There are no end of other software products which can serve up views with more than 5,000 rows. If there is technical issue preventing this in SharePoint Online, then it needs to be addressed.
Andrew Armanini commented
This is a show stopper for even a small office (around 50 users). If you read the "how to deal with large lists" article, it's a catch-22. The only way to get large data in is via creating a list by uploading an Excel spreadsheet. That method only allows the creation of a new list with custom columns. You can't retroactively add indicies and searches to custom columns. So you have an unusable list. And this is without thinking about a document library which excel can't deal with.
This requires the use of third party tools with subscriptions in the thousands of dollars a month, because there's no inbuilt way to migrate list items.
SharePoint Online as it stands is fundamentally broken. You can have 30 million items, as long as there's no more than 5000. SharePoint can do amazing things, except if you actually try to use it.
We at the very least need to be able to retroactively add indexes and switch a custom column to an inbuilt indexable column. It's too easy for a client to work their way into the threshold 5000 trap - with no way out of it.
Martin Coupal commented
Why not enhance and leverage the search engin? You should provide a way to create list/library views based on search and scoped to the list where the results would be presented in a similar layout than a list view (ECB menu etc.).There should be a way to provide real-time search results by using a kind of event driven indexing. This would probably be easier than change how sharepoint stores list data in SQL.
Clients who have worked on prem in a certain way multiple sorts and filters no matter how many indexed columns you have this will not work in o365 with the limits in place , this in turn results in clients looking at other solutions
It's 2017. 5,000 item threshold is just insane. Right now, I can't even setup a workflow to do a lookup against a list with 5,000+ items.