Remove the list view threshold (5000 by default)
This limit has always been a bit laughable, and is even more so as we develop more client side applications. In SharePoint 2007 we didn't have this limit and were allowed to make our own mistakes. Now that hardware is so much more powerful, we need this limit removed so that we can build enterprise-class applications.
We are continuing to make our large list experiences better, please keep the feedback coming.
Spring 2018 update:
- We now support being able to manually add indexes to lists of any size (increased from lists up to 20,000 items previously).
- Starting with the February release of the Office 365 Excel client, you will be able to export your full list instead of getting cut off part of the way through.
What we are working on now:
- Predictive indexing will start to work for lists larger than 20,000 items so your views will automatically cause the right indexes to be added to your lists.
In our backlog:
- Being able to index/sort/filter by lookup column types (like person, lookup or managed metadata columns) without being throttled.
- Making sure that our REST APIs support querying in ways that will guarantee that the call will not be throttled.
For a general update on large list capabiltiies, the video on myignite.microsoft.com/videos/53861 (focusing on large lists at 42 minutes and 25 seconds) describes some of the changes that we delivered back in the second half of 2017:
- Modern UI now has a lot of support for adding indexes to large lists and libraries on the fly, reducing the number of throttling errors experienced by our users, and some new UI for browsing through items in large lists with our paging model
- SharePoint runs predictive indexing jobs to automatically add indexes as lists get larger based on your view definitions, and updates these indexes when you add/update your views
Looking forward to receiving more of your feedback.
Our customers are "thinking about it" too... thinking about leaving the platform. Please for starters at least allow us to index lists that are over the limit without having to delete or migrate data out. Perhaps the underpinnings of the Custom List need to be completely revamped. 5000 is an embarrassing limit we just can't explain to customers.
Maybe if it was easier to apply a filter on the required data to only show items beginning with a range, e.g. items starting with A-C, D-F etc. Or depending on the size of the list, even just a single character. This could help get the results below the magic 5,000 limit. I don't just mean a filter for viewing purposes, what if you want to query the data let's say via a lookup column for applying metadata to uploaded documents for e.g.
I think that would be helpful...
But also not being able to index columns that have exceeded 5 000 is just silly!
if your document library has over 5000 items you cannot even create indexes as it errors. Rubbish product
I highly recommend that alternate products should be used, like "Confleuence". SharePoint is dead in the water.
The 5000 item view limit is one thing, but what really makes life hard is that there's no way to even start filtering without an initial view under 5k. If we have 25k items in a list, the first screen should say "that's too many to display. start filtering to get below 5k" From there it's easy to get under the limit. However, the only option is to pre filter the 25k to smaller, arbitrary subsets, which then makes subsequent filters incomplete.
another idea would be to allow us to take a view under 5k, further refine it, then take away the filters that allowed the first view to proceed, as long as the result is still under 5k. In this way, we can have a filtering interface for the entire library.
The list view threshold is to maintain performance so that SQL tables aren't locking upon many concurrent requests. Why not some guidance when increasing the list view threshold is acceptable depending on the amount of SQL row locking occurring based on list view queries? I know of some SP deployments where the list view threshold was increased to 20,000 for 2000 users and probably 70 concurrent user sessions during business hours and performance seemed fine.
Andy Tous commented
No, that does not sound reasonable. Your answer might sound logic for an average joe lemon stand, but not for a Fortune 300 company with tons of information on SharePoint. Here's why:
Microsoft support states that SharePoint lists can hold 30 million records. 30 MILLION RECORDS!
Not being able to display more than 5000 items at a time in a view is understandable and logical in order to maintain performance. What is not reasonable at all, thus becoming an impossible task to manage, is the facs that, once you pass the 5000 records limit, you're unable to create filtered list views that return fewer than a hundred results (will not even returns just a few records). THis happens even if the fields being called in the view are indexed.
I tried to create a view in a library of 6000 items that would return 100 records. Nope. Can't do that.
But wait! If I use All Items views, that works! How is that logical?
How in the world would you expect a list that should hold 30 million records to be navigable or manageable without the ability to create views that filter, sort and group?
Your sales and support are both misleading and should state that the limit for SharePoint lists is just 5000 records.
What a joke!
Absolutely DISGUSTED with Microsoft over this issue. First of all, if you can't handle more than 5,000 files, why would you ever sell this product to a business? And second, it is completely unreasonable to expect a small or medium business to have the resources or expertise to figure out how to "get past this limitation". The whole point of your product is to make things easier, but we now have hundreds of hours invested into figuring out why 2 out of 8 users cannot sync. Just 2. Not all 8. If this is truly a legitimate issue, how the other 6 work fine is a mystery to me. Our business has been unable to function for months. We use other products that cannot communicate with SharePoint directly; so syncing is an essential feature and one of reasons we selected OneDrive in the first place. If you aren't going to fix the problem, you need to prominently disclose it as a prohibitive limitation so that you don't trick small businesses into investing in a product that doesn't meet their needs.
The problem is that this LIMIT affects other simple display -- for instance, the method to setup the default OneNote Notebook and Section to use for emails sent to onenote (firstname.lastname@example.org) (https://www.onenote.com/EmailSettings) is restricted by this list view threshold limit (as documented here: https://answers.microsoft.com/en-us/msoffice/forum/msoffice_onenote-mso_mac/notebook-retrieval-error-in-onenote-online/fd41bc98-8537-4df9-8e1e-c15c71bec206).
This doesn't make sense, since OneNote Notebooks are their own thing and there is no need to list the rest of the files in my OneDrive.
Changing how we use OneDrive fundamentally reduces its utility. I cannot reduce my number of files as many are created as part of automated analysis scripts, etc.
Here's the issue. SharePoint lists are supposed to hold 30 million files. 30 MILLION. I understand not being able to display more than 5000 items at a time. What I don't understand is that once you're over 5000 files, you're unable to create filtered list views that return way fewer than 5000 results. I tried to create a view in a library of 13000 items that would return 16 files. Nope. Can't do that. How in the world would you expect a "small" library of 1 million files to be navigable without the ability to create views to filter, sort, group, etc. You may as well say that the limit for SharePoint libraries is 5000 files.
Not reasonable at all!
Noah Stahl commented
I don't think the proposal is reasonable. This is as others have pointed out a fundamental blocker that can't realistically be overcome by anticipating a limit of records in a given view. I think this should be top priority to solve. Surely the technical capability and infrastructure is there to make this possible.
San Mattel commented
5000 default limit is really frustrating. Many cases client not ready to take a risk by increasing this limit and looking at other solutions
Russell Gove commented
If The 5,000 item threshold is likely here to stay, then we need to have better tools to manage it. I dont know I have an issue until a user calls and reports an error. At that time it;'s often too late to provide a reasonable solution.
It would be nice if an admin could set the listview threshold to a lower limit (say maybe 4000. Then, when a user calls to report an issue the admin can bump it up to 5000 to get the users up and running, then add indexes, create views, whatever to resolve the issue, and then set it back to the lower limit. This would give us advancred notice and the ability to test if our workarounds actually work
Jim K commented
Microsoft seems to be positioning sharepoint online as a google drive, box, dropbox competitor - i.e. a file server replacement. None of these other services have this ridiculous restriction. It's a deal killer for using sharepoint online as a file server replacement. Which, in turn, makes us want to look at other solutions such as g suite.
The response by the SharePoint Experiences Team seems to reflect an upside down view of the world. Technology is subservient to people, not the opposite. Asking list owners and users to work around, or against, limits not justified by current technology is not reasonable, and neither is expecting users to think like developers.
Balu Dharmarajan commented
This limit is quite low, I already checked the dll codes and sql queries and it looks like not too complicated to fix, please let me know I will tell you how it can be fixed.
At the Virtual SharePoint Summit, on a slide titled 'Transform business process roadmap', MS announced 'By year end: intelligent indexing for large lists'. Please MS don't forget about any column type: https://ktnnsharepoint.wordpress.com/2016/10/22/column-types-list-view-threshold/
Pramod Nathan commented
Remove List view threshold limit of 5000 by taking advantage of paging
The current list view threshold limit of 5000 with it's workarounds are very frustrating. We can all appreciate that we do not want to fetch too many items from the table (the underlying cause of the restriction). BUT paging exists and Microsoft needs to take better advantage of this:
1. The SharePoint interface is page based. Instead of breaking (which is what the solution does with a warning), MS can change the underlying call so it breaks down the call to SQL to fetch data as needed per page/set of pages. When the user navigates to the next page/set of pages - call SQL for the next set of data (e.g. if I am on the current view with 5000+items, I really don't care about about item 5000+1 immediately. BUT I do care that I can manipulate the entire set of items and that I can get to item 5001). So just fetch the first few page worth of items.
2. For sorting and filtering, the call can go to SQL on click to fetch the top X number of items depending on the filter or sort. (e.g. if the page has 5000+items and we want to sort by a column, it can call a SQL statement that does this but sends back only the top 3-4 pages or how many ever are feasible)
3. It will also make the underlying APIs very robust in terms of being able to create multiple calls of data to generate a large data sets that are stored in SharePoint lists and libraries.
the 5,000 is pushing us to pursue other solutions outside SharePoint.
We created a list to hold user's stats; and for stats purposes it is worthless to retrieve chunks of data when you want to know how many selected a, b, or c, or group them by hour.
As a workaround, we connected the list to PowerBI. From there we created a dashboard which actually can work with all the data at the same time. Until now, this workaround worked just fine, but we just got a message saying that dashboard sharing will no longer be part of PowerBI free.
We just want SharePoint to work and not to start thinking on workarounds or 3rd party applications to make it effective and efficient.