=========================================================================== IMC 2010 Review #281A Updated Friday 18 Jun 2010 2:43:12am EDT --------------------------------------------------------------------------- Paper #281: On Economic Heavy Hitters: Shapley value analysis of the 95th-percentile pricing --------------------------------------------------------------------------- ===== Paper summary ===== Access link providers typically charge a fixed cost to customers while incurring variable (95th percentile) transit costs. This work considers the intuitive notion of "the cost of a user's bytes is higher during peak usage times" by distributing costs with the Shapley notion of fairness. Develops, and evaluates the accuracy of, a MC based method to determine the Shapley value for each user across a large number of users (required because of the factorial number of coalition permutations). Using this method, evaluates the behavior and attributable cost of 10K users over a month period on an access network. Novelty: 2. New contribution ===== Exciting ===== Applies a well-known method of attributing (value/cost) to the large-scale problem domain of individual users. In addition, estimation errors are considered. The method is applied to a real dataset to gain some interesting intuition about the usage behavior of users relative to their induced cost. ===== Run of the mill ===== Would be nice to have additional discussion on the relevance of the work, of which I think much can be said (and the work is very relevant). I.e. the work is valuable but if users demand a fixed pricing model, how will this model apply -- how can providers use this to discriminate or provide differentiated pricing models akin to the airlines? It's out of scope for a short paper to consider in detail, but some comparisons to e.g. congestion-based pricing would add value. ===== Limitations ===== The statement that "even if a user does not generate any traffic in the peak hours does not imply that its impact toward the 95th percentile is zero" need to be better justified, as this is true only in the context of Shapley value. ===== Comments for author ===== 1. The differences across users between Shapley value and their aggregate contribution is very interesting. But since the majority of probability mass is (Figure 5) is around 1, does this imply that pure usage-based billing suffices? Some additional commentary on this would be valuable. 2. Because of quantization effects of the histogram binning, it would be interesting to see 1/p_i in addition to Figure 5. 3. Last paragraph of section 3.1 needs better justification. In particular, given the previous argument that users sending not at peak usage hours should contribute to the marginal contribution of the 95th percentile, so too should the traffic in the upstream direction. =========================================================================== IMC 2010 Review #281B Updated Thursday 24 Jun 2010 5:24:16pm EDT --------------------------------------------------------------------------- Paper #281: On Economic Heavy Hitters: Shapley value analysis of the 95th-percentile pricing --------------------------------------------------------------------------- ===== Paper summary ===== The authors evaluate the impact of a 95th-percentile policy that determines the ISPs cost to their transit provider. Because of this policy users do not contribute to the ISP’s cost in a manner that is proportional to the total traffic they inject into the network. The authors show that instead traffic sent during peak-hour periods effectively costs more. They capture this effect by using the “Shapley value” metric to quantify a user’s contribution to an ISP’s costs. Using 30 days of data from 10K users, they show that for roughly 10% of the users, their cost contribution is twice that of their byte contribution, and similarly for roughly 10% of users, their cost contribution is half that of their byte contribution. Their purpose is not to propose an alternate pricing scheme but rather to point out the effects of the 95th-percentile (burstable billing) policy that transit providers use to charge their ISP customers. Novelty: 2. New contribution ===== Exciting ===== The authors propose a metric that captures the inequity between a the volume of traffic an ISP carries for a user and the cost the ISPs incurs for each user. They illustrate the inequity arises due to the burstable billing policy. It is nice to quantify this discrepancy that results from seemingly simple policies today, so that someday in the future it might lead to fairer billing policies. ===== Run of the mill ===== The authors don’t interpret the results enough to assist the reader. Sections 3.3 and 3.4 need more discussion. You don’t actually conclude as to whether or not the numbers in the graphs mean that the current policy is bad or not (one could argue that perhaps it isn’t so bad – see below). ===== Comments for author ===== So what’s your conclusion in Sections 3.3 and 3.4. Is this grossly unfair? Is 10% of \rho>2 and 10% of \rho<1.2 really a bad thing? Based on the graph in Figure 4, it seems most users are not too far from the y=x line, meaning that the inequity for those users isn’t that bad. If the balance between shapley value cost and byte count is similar for the majority of the users, then perhaps one could argue that burstable billing isn’t really that bad. In section 3.4 could you explain more the following? The peak period is only 2 hours, but you say that the cost is positive for 6 hours. Why is that? Is it because during those 6 hours the traffic was large enough to form the “middle” part of the distribution, thereby pushing the peak hours into the range of a 95th percentile? There is no need to have an appendix here. The material is short and simple, and should go into the paper (probably into section 1). Why not go the next step and talk about how billing could be improved ? =========================================================================== IMC 2010 Review #281C Updated Friday 9 Jul 2010 10:31:19am EDT --------------------------------------------------------------------------- Paper #281: On Economic Heavy Hitters: Shapley value analysis of the 95th-percentile pricing --------------------------------------------------------------------------- ===== Paper summary ===== This paper uses Shapley value as an analysis tool for understanding the impact of individual users on an access provider's costs. Novelty: 2. New contribution ===== Exciting ===== Shapley value as a metric -- rather than as part of an incentive mechanism -- is an interesting perspective. The paper delivers insights on a problem that is hard to formalize. ===== Comments for author ===== This paper uses Shapley value as a tool for understanding the impact of individual users on an access ISP's uplink costs. This is a timely and important question. This is not the kind of paper that defines a crisp problem and then proceeds to a demonstrable solution. Rather it seeks to measure, for the purposes of insight, a phenomenon that is hard to quantify: the impact of an individual user on an ISP's costs. The problem is hard to quantify because user impacts are not additive or easily separable: the impact of any given user depends on the actions of many or all other users. In this context, the proposal to use Shapley value as an analytic tool is a significant contribution in my view. The authors make a convincing argument that Shapley value is a relevant metric for this question, and they propose a sampling method for estimating it in the case of many users. The results, as might be expected in a short paper, are somewhat preliminary, but they show that the metric has potential to yield insight. In particular, it shows that a small subset of users have a disproportionate impact on an ISP's costs (note that this was already understood heuristically, but not in any particularly formal way) and that users' actions in only a small portion of the daily cycle are relevant to an ISPs costs (again, folk wisdom but not previously easily-quantifiable). I expect that this paper would stimulate further research using Shapley value as an analytic tool. =========================================================================== Comment Paper #281: On Economic Heavy Hitters: Shapley value analysis of the 95th-percentile pricing --------------------------------------------------------------------------- This paper was "pre-accepted" before the PC meeting and therefore no meeting notes are available.