From annc@ISI.EDU Thu Sep 5 01:21:32 2002 From: annc@ISI.EDU (Ann Chervenak) Date: Wed, 04 Sep 2002 17:21:32 -0700 Subject: [Rls-internal] RLS issues Message-ID: <5.1.0.14.2.20020904171727.0b6fcdf0@tnt.isi.edu> Hi, I have created this new rls-internal mailing list because, as long as the EDG folks continue to be completely disengaged from design, development and testing, I think there are going to be an increasing number of issues that we will need to discuss independently of them. I talked briefly to Peter Kunszt today on another topic, and I took the opportunity to ask him about the status of their testing. He was evasive, but it is pretty clear that they have not made any progress. He indicated that the EDG applications people are strongly demanding a production-quality EDG testbed, and that they are increasingly resistant to putting new software (e.g., the RLS) on the EDG testbed. So, it seems that the chances of the RLS software being widely deployed in Europe in time for the SC2002 demo are slim. I tried to verify with him that at least the 6 to 8 machines that they had used in the past for RLS testing would be available to us for the SC demo. This seems likely. I also suggested that we set up a phone call for next week to discuss plans for the SC2002 demo of the RLS. This will most likely happen on Tuesday morning 8am PT/10am CT. Bob and I had a general discussion about RLS development on Tuesday, and I would like to invite feedback on several issues that Bob raised: 1) With the current implementation of bloom filter compression, there is no way to do wildcard searches on an RLI. Since wildcard searches are supported in the RLI API, this seems to "break" the RLI from the user's perspective. One thing that Bob suggested was that if the user issues a wildcard search to the RLI, the RLI could dispatch the corresponding query to the LRCs that send it state updates, gather the results of those LRC searches and return them to the user. This would be slower than wildcard searches without bloom filters, but it wouldn't cause the wildcard search on the RLI to return an error that the command is unsupported. Comments on whether this is the right thing to do? 2) When Bob implemented bloom filters, he removed the partitioning code for the RLIs. I would argue that we need to put the partitioning back. One argument for partitioning was to reduce the amount of data and the network traffic that was required for soft state updates. Bob's intuition was that these were sufficiently reduced by the bloom filter compression that partitioning was no longer necessary. However, I think there are other good reasons for partitioning, including the ability to provide customized index nodes that contain only a portion of the logical file namespace. Bob was motivated to remove the partitioning because of the large memory overhead associated with maintaining the information required to create one bloom filter per RLI. We agreed that we could save some of this memory by maintaining a bloom filter per partition rather than per RLI, and I think that is what we will probably do. Are there other opinions about whether partitioning is still necessary when used in combination with bloom filter compression? 3) We discussed whether it makes sense for Bob to implement consistent hashing as a partitioning strategy. Bob says that the implementation would be trivial. However, I have never been quite convinced that consistent hashing on a grid that spans the wide area is a good idea. My understanding is that consistent hashing will spread the LFN mappings among the RLIs using the hash function, assuming that all RLIs are basically equivalent, regardless of whether some of them are situated locally on a high-performance network or over a slow trans-Atlantic link. Is this correct? Does anyone know of any algorithms for weighting the hash or specifying classes of RLIs based on their likely performance? Should we implement consistent hashing as one potential partitioning strategy that can be used, for example, in well-connected environments? Thanks for your input. Ann From bobs@ISI.EDU Thu Sep 5 02:06:05 2002 From: bobs@ISI.EDU (Robert Schwartzkopf) Date: Wed, 04 Sep 2002 18:06:05 -0700 Subject: [Rls-internal] RLS issues In-Reply-To: Your message of "Wed, 04 Sep 2002 17:21:32 PDT." <5.1.0.14.2.20020904171727.0b6fcdf0@tnt.isi.edu> Message-ID: <200209050106.g85165d27877@nitro.isi.edu> >1) With the current implementation of bloom filter compression, there is >no way to do wildcard searches on an RLI. Since wildcard searches are >supported in the RLI API, this seems to "break" the RLI from the user's >perspective. One thing that Bob suggested was that if the user issues a >wildcard search to the RLI, the RLI could dispatch the corresponding query >to the LRCs that send it state updates, gather the results of those LRC >searches and return them to the user. This would be slower than wildcard >searches without bloom filters, but it wouldn't cause the wildcard search >on the RLI to return an error that the command is unsupported. Comments on >whether this is the right thing to do? Laura suggested implementing this in the client API rather than in the RLI server, which seems like a good idea, should we do this. Specifically if the API does a wildcard search and gets back an "unsupported due to bloom filter" error, it then automatically retrieves the list of LRCs known by the RLI, and does the wildcard search on each of those in turn. This has the advantage of pushing this work down to the client host rather than making the RLI server do it, while still hiding the complexity from the user. Re points 2, 3 (global filter vs partitioned filter, and consistent hashing), first Ann said something I wanted to clarify. I didn't remove the existing partition code, it's just ignored when a bloom filter is used to update an RLI. Also wanted to stress again that the only reason to partition bloom filters is to be able to create RLI servers that only know about a subset of the LFNs (presumably for security reasons?). There's no significant performance gain, and the LRC code gets more complicated and may need considerably more memory if there are filters for each partition (depending on how it's implemented and how much overlap there is between partitions). So we need to decide if hiding the existence of an LFN from an RLI is worth the cost of supporting this. Note that if consistent hashing is used for partitioning, then the ability to control which RLIs know about which LFNs is lost, so there's definitely no point in maintaining separate filters. Also the cost of maintaining the filter may be high if the available RLIs at any point is dynamic, since whenever an RLI comes up or disappears all the bloom filters have to be recomputed. In fact I think the notion of consistent hashing, combined with bloom filters, doesn't make much sense. I would suggest that in this case consistent hashing be modified to mean send the global filter to some number of available RLIs as opposed to send each partition specific filter to some number of available RLIs Bob Ps. If you're curious the performance tradeoffs with partition specific filters are as follows. Presumably the partitioned filters are smaller than the global filter, and hence would take less time to transmit to an rli. This is a good thing, but there are 2 points to consider: 1. Transmit time for a million record bloom filter is essentially zero (less than half a second over fast ethernet). So this is only going to help if the partition drastically shrinks the bloom filter, and you're updating over a slow (eg T1 speed) link. If a filter is 10 times the number of LFNs then a million LFNs means a 10 megabit filter. Even over ethernet that's only a few seconds, and over T1 it's probably less than 30 seconds. 2. The LRC will have difficulty computing the optimal size of a partitioned bloom filter. Currently it counts the number of LFNs in the LRC at start time, and sizes the global bloom filter appropriately. And as LFNs are created it automatically recreates the bloom filter whenever the number of LFNs increases beyond a certain amount (currently whenever the LFNs double). If there are partition specific bloom filters I either continue the above (simple) method, which means all partitioned bloom filters are just as big as the global one (meaning no performance gain at all), or I try to maintain the size of each filter separately. This would increase the work the LRC has to do as LFNs are added/deleted, since it would need to match each LFN against all the partitions and update filters appropriately (including resizing when necessary). From matei@cs.uchicago.edu Thu Sep 5 20:17:05 2002 From: matei@cs.uchicago.edu (Matei Ripeanu) Date: Thu, 5 Sep 2002 14:17:05 -0500 (CDT) Subject: [Rls-internal] RLS issues Message-ID: <20020905191705.627942AB039@buda.cs.uchicago.edu> some comments below... > > Re points 2, 3 (global filter vs partitioned filter, and > > consistent hashing), > > first Ann said something I wanted to clarify. I didn't remove the > > existing partition code, it's just ignored when a bloom filter is used > > to update an RLI. > > > > Also wanted to stress again that the only reason to partition > > bloom filters is to be able to create RLI servers that only know about > > a subset of the LFNs (presumably for security reasons?). There's no > > significant performance gain, and the LRC code gets more complicated and > > may need considerably more memory if there are filters for each partition > > (depending on how it's implemented and how much overlap there is between > > partitions). > > > > So we need to decide if hiding the existence of an LFN from an RLI is > > worth the cost of supporting this. If this is the level of granularity maybe there are simpler solutions: do not send to that RLI the filter coresponding to the LFN you want to hide (this assumes that RLI have a list of pairs