This is a growing and shrinking list of known barriers to scalability. Have no fear, ZipTie scales very well at present. But the goal is to support 500,000 devices in essentially the same operational memory footprint as 500 devices. At times, for expediency we have made calculated trade-offs to provide functionality to our users sooner. This page essentially captures those trade-offs, assesses their impact, and discusses remediation.
ZipTie provides an extension mechanism by which developers can add new types of device IP resolvers. For example, an IP Resolver which given a 'tag' returns the list of devices (IP addresses) that have that tag. All IP Resolution Schemes (aka IP Resolvers) return a java.util.List of ZDeviceLite objects. Clearly, if the resolution requires returning 500,000 devices then this would result in 500,000 objects being created in memory.
The IP Resolvers should return a PageData object whereby devices can be incrementally retrieved from the database. By doing so, we ensure the same footprint for resolution for small and large device counts.
The solution has been implemented but the jobs and various other places need to be changed to take advantage of it.
Everywhere that uses DeviceResolutionElf#resolveDevices needs to be updated to use the paged API on the same Elf.